I am trying to parse string from a list of sentences using regex search.
The list of sentences that I have:
- my name is 'Andi', nice to meet you
- hello my name is 'Bella', hi!
- this is 'Cain'
The output that I want:
- Andi
- Bella
- Cain
What I have tried:
- set_fact:
test: >-
{{
sentences | map("regex_search", "is '(.*)'", "\1") | flatten
}}
vars:
sentences:
- my name is 'Andi', nice to meet you
- hello my name is 'Bella', hi!
- this is 'Cain'
This is gives me what I want which is:
- Andi
- Bella
- Cain
My problem is when my data is like this:
- my name is 'Andi', nice to meet you
- hello i am me!
- hello my name is 'Love', hi!
- this is 'Cain'
I expected it to show result like:
- Andi
- Love
- Cain
But instead it skipped everything after the unmatched string and show results like this:
- Andi
How to skip only the unmatched line instead of everything ?
In a nutshell, the following sentence.yml playbook:
---
- hosts: localhost
gather_facts: false
vars:
sentences:
- my name is 'Andi', nice to meet you
- hello i am me!
- hello my name is 'Love', hi!
- this is 'Cain'
name_regex: "^.*is '(.*)'.*$"
names: '{{ sentences | select("regex", name_regex) | map("regex_replace", name_regex, "\g<1>") | flatten }}'
tasks:
- debug:
var: names
gives:
$ ansible-playbook sentence.yml
PLAY [localhost] **************************************************************************************************************************************************************************************************
TASK [debug] ******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"names": [
"Andi",
"Love",
"Cain"
]
}
PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Related
--- EDIT ---
Partial-Solution: Messed around with the JMESPath syntax and was able to successfully get a match for the first test case (without the optional variable) using:
jmesquery: "{{ datacenter }}{{ subcategory }}.{{ refine_hosts }}.[*][].[*][][]"
I am writing an Ansible Playbook that takes a list of hosts from a network server, parses the JSON list, and finds hostnames that matches the user's input when they deploy the playbook as a Jenkin's Job through it's API.
The issue I am encountering is that I am unable to successfully query the JSON host list. Currently, I am only trying to run the following test case:
datacenter: a
subcategory: bc
refine_hosts: QA
However, the final version of this playbook should be able to take in values for datacenter, subcategory, and refine_hosts with an optional input value of host_type. An example test case including the optional input value would be the following:
datacenter: a
subcategory: bc
refine_hosts: QA
host_type: WEBSITE
In my playbook, I am using JMESPath within the following task:
- name: Build HOSTS list
set_fact:
hosts_list: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: '%%datacenter%%-%%subcategory%%.%%refine_hosts%%.[*][*][][]'
The JSON host list is structured in the following manner (I am unable to edit the structure of the host list, but it will always follow the following structure nonetheless):
{
"a-bc":{
"all":{
"webServer":[
],
"archive":[
"someHostAlias-123.privateDomain.com"
],
"central":[
"someHostAlias-456.privateDomain.com"
]
},
"QA":{
"xyz":{
"INBOUND_HTTP":[
"someHostAlias-789.privateDomain.com"
],
"WEBSITE":[
"someHostAlias-1011.privateDomain.com"
]
}
}
}
}
I have been using the following websites for this issue:
JMESPath Tutorial
Ansible JMESPath Documentation
JSONPath Expression Tester
StackOverflow: How to Use Variable in JMESPath Expression
Gitter: JMESPath/chat
I apologize if the query seems obvious, this is my first attempt at an Ansible Playbook. All help/feedback is greatly appreciated.
One of the issue of your query is that you are confusing [*] — a list projection — that selects all the elements of a list with .* — an object projection — that selects all the properties of a dictionary.
So, one solution in JMESPath, would be to do:
jmesquery: >-
"{{ datacenter }}-{{ subcategory }}".{{ refine_hosts }}.*.
{{ host_type if host_type | default('') != '' else '*' }}[] | []
Given the playbook:
- hosts: localhost
gather_facts: no
tasks:
- debug:
msg: "{{ jsondata | json_query(jmesquery) }}"
loop: "{{ fake_user_input }}"
loop_control:
label: "{{ jmesquery }}"
vars:
jmesquery: >-
"{{ datacenter }}-{{ subcategory }}".{{ refine_hosts }}.*.
{{ host_type if host_type | default('') != '' else '*' }}[] | []
datacenter: "{{ item.datacenter }}"
subcategory: "{{ item.subcategory }}"
refine_hosts: "{{ item.refine_hosts }}"
host_type: "{{ item.host_type | default('') }}"
fake_user_input:
- datacenter: a
subcategory: bc
refine_hosts: QA
host_type: WEBSITE
- datacenter: a
subcategory: bc
refine_hosts: QA
jsondata:
a-bc:
all:
webServer: []
archive:
- someHostAlias-123.privateDomain.com
central:
- someHostAlias-456.privateDomain.com
QA:
xyz:
INBOUND_HTTP:
- someHostAlias-789.privateDomain.com
WEBSITE:
- someHostAlias-1011.privateDomain.com
This yields:
ok: [localhost] => (item="a-bc".QA.*. WEBSITE[] | []) =>
msg:
- someHostAlias-1011.privateDomain.com
ok: [localhost] => (item="a-bc".QA.*. *[] | []) =>
msg:
- someHostAlias-789.privateDomain.com
- someHostAlias-1011.privateDomain.com
I have a functioning ansible script that connects to AWS and prints out tags from instances in a specified state.
The problem I'm having is the print out is two separate lists e.g.
What I get is:
name1
name2
name3
description1
description2
description3
What I would like is:
name1 - description1
name2 - description2
name3 - description3
I've tried pushing this all into a dictionary, but got lost in the attempt. There must be an easier way.
Here's my code:
- name: print stopped systems
hosts: all
become: false
tasks:
- name: Gather ec2_metadata_facts (use -vv to show all)
action: ec2_metadata_facts
- name: pull instance info with ec2_instance_info
ec2_instance_info:
region: "{{ lookup('env','AWS_DEFAULT_REGION') }}"
aws_access_key: "{{ lookup('env','AWS_ACCESS_KEY_ID') }}"
aws_secret_key: "{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}"
filters:
instance-state-name: [ "shutting-down", "stopping", "stopped" ]
register: ec2_info
- name: print Instance Info
debug:
msg:
- "{{ ec2_info | json_query(name_tag) }}"
- "{{ ec2_info | json_query(description_tag) }}"
vars:
name_tag: "instances[*].tags.Name"
description_tag: "instances[*].tags.Description"
The output of the debug statement looks like this:
TASK [print Instance Info] **********************************************************************************************************************
ok: [tools-server-01] =>
msg:
- - server-01
- server-02
- - Description for server one
- Description for server two
Thanks #Rickkwa
I've got this mostly working.
here's my current code
- name: Print stopped systems
hosts: all
become: false
tasks:
- name: Gather ec2_metadata_facts (use -vv to show all)
action: ec2_metadata_facts
- name: pull instance info with ec2_instance_info
ec2_instance_info:
region: "{{ lookup('env','AWS_DEFAULT_REGION') }}"
aws_access_key: "{{ lookup('env','AWS_ACCESS_KEY_ID') }}"
aws_secret_key: "{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}"
filters:
instance-state-name: [ "shutting-down", "stopping", "stopped" ]
register: ec2_info
- name: Create Name list
no_log: true
set_fact:
name_l: "{{ec2_info | json_query(name_tag)}}"
vars:
name_tag: "instances[*].tags.Name"
- name: Create Description list
no_log: true
set_fact:
desc_l: "{{ ec2_info | json_query(description_tag) }}"
vars:
description_tag: "instances[*].tags.Description"
- name: print together
debug:
msg: "{{item.0}} --- {{item.1}}"
loop: "{{ name_l|zip(desc_l)|list}}"
Unfortunately, the last debug statement includes some 'garbage" I'd like to remove...anyone have a suggestion to get rid of the stuff following the "u"s?
here's the output :'
TASK [print together] ***************************************************************************************************************************
ok: [dev-bsd-01] => (item=[u'server-01', u'server one description']) =>
msg: server-01 --- server one description
ok: [dev-bsd-01] => (item=[u'server-02', u'server two description']) =>
msg: server-02 --- server two description
I found this page from answer number 4 by #cobbzilla useful to my use case.
Just want to ask if Ansible is capable to have a function that will handle this command:
2>&1 >> /tmp/debug.log
I have already applied this solution to my yml files and I was looking if the command can be wrapped on a function so that it will show much cleaner.
My Sample Ansible yml:
- name: Deploy
hosts: localhost
connection: local
gather_facts: false
tasks:
- name: perform 1st script
shell: bash -c "1st_script.sh 2>&1 >> /var/tmp/debug.log"
- name: perform 2nd script
shell: bash -c "2nd_script.sh 2>&1 >> /var/tmp/debug.log"
- name: perform 3rd script
shell: bash -c "3rd_script.sh 2>&1 >> /var/tmp/debug.log"
Preliminary note: I am answering your direct question below because this can be useful in other circumstances.
Meanwhile, in the ansible context, playing bash scripts and returning their output and error to a log file on the target is generally not a good idea. You will be left blind if something goes wrong, or if you want to analyze the output, as the module will return an empty stderr and stdout. You will have to rely on analyzing the log file later (and finding the correct output since you mix script output in there).
On an even wider level, you should use shell/command only when there is no other possibility to get the same job done using existing modules
If you don't mind writing some lines of python, a pretty easy and straight forward way to acheive your goal is to use a custom filter. The below example is quickNdirty. You will probably have to harden its code (escape command characters, check for specific errors...) but this should put you on track.
For the example, I am creating the filter in the filter_plugins folder adjacent to the demo playbook. If you need to distribute that filter across projects, search the ansible documentation to learn how to wrap that in a role or a collection.
In filter_plugins/shell_filters.py
def bash_n_log(command, log_file='/var/tmp/debug.log'):
"""Return a formatted string to play the script in bash and log its output"""
return f'bash -c "{command} 2>&1 >> {log_file}"'
class FilterModule(object):
"""collection of shell utility filters."""
def filters(self):
"""Return the filter list."""
return {
'bash_n_log': bash_n_log
}
Then the demo playbook.yml
- name: Custom shell filter demo
hosts: localhost
gather_facts: false
vars:
my_commands:
- echo Hello World
- ls -l /dev/null
tasks:
- name: Play my commands with my filter
shell: "{{ item | bash_n_log }}"
loop: "{{ my_commands }}"
- name: Same example with non default log file
shell: "{{ item | bash_n_log('/tmp/other.log') }}"
loop: "{{ my_commands }}"
- name: Get content of the log files
slurp:
path: "{{ item }}"
register: slurped_logs
loop:
- /var/tmp/debug.log
- /tmp/other.log
- name: Show log file content
debug:
msg: "{{ (item.content | b64decode).split('\n') }}"
loop: "{{ slurped_logs.results }}"
loop_control:
label: "{{ item.item }}"
gives:
$ ansible-playbook playbook.yml
PLAY [Custom shell filter demo] ********************************************************************************************************************************************************************************************************
TASK [Play my commands with my filter] *************************************************************************************************************************************************************************************************
changed: [localhost] => (item=echo Hello World)
changed: [localhost] => (item=ls -l /dev/null)
TASK [Same example with non default log file] ******************************************************************************************************************************************************************************************
changed: [localhost] => (item=echo Hello World)
changed: [localhost] => (item=ls -l /dev/null)
TASK [Get content of the log files] ****************************************************************************************************************************************************************************************************
ok: [localhost] => (item=/var/tmp/debug.log)
ok: [localhost] => (item=/tmp/other.log)
TASK [Show log file content] ***********************************************************************************************************************************************************************************************************
ok: [localhost] => (item=/var/tmp/debug.log) => {
"msg": [
"Hello World",
"crw-rw-rw- 1 root root 1, 3 Mar 27 12:06 /dev/null",
""
]
}
ok: [localhost] => (item=/tmp/other.log) => {
"msg": [
"Hello World",
"crw-rw-rw- 1 root root 1, 3 Mar 27 12:06 /dev/null",
""
]
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
I just tried and tested this simple solution worked, I just added vars section witch points the variable to a command
- name: Deploy
hosts: localhost
connection: local
gather_facts: false
vars:
logger: "2>&1 >> /var/tmp/debug.log"
tasks:
- name: perform 1st script
shell: bash -c "1st_script.sh {{ logger }}"
- name: perform 2nd script
shell: bash -c "2nd_script.sh {{ logger }}"
- name: perform 3rd script
shell: bash -c "3rd_script.sh {{ logger }}"
How do you remove a json Key that contains a colon (:) with jinja2 rejectattr.
Environment:
ansible 2.9.1
config file = None
configured module search path = [u'/home/<user>/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.15+ (default, Oct 7 2019, 17:39:04) [GCC 7.4.0]
json data:
{
"tag:environment": "qa",
"tag:instance_id": "i-123456789"
}
Ansible playbook:
- name: Remove InstanceID
debug:
msg: "{{ instance_filter | rejectattr('['tag:environment'], 'defined' ') | list }}
Actual Results:
fatal: [localhost]: FAILED! => {
"msg": "template error while templating string: expected token ',', got 'tag'. String: {{ instance_filter | rejectattr('['tag:environment'], 'defined' ') | list }}"
}
Expected results:
{
"tag:environment": "qa"
}
The rejectattr is indeed one of the key filters to use to achieve your goal, but a few more things are needed. Here is the correct sequence of filters to remove that particular key from the dictionary variable you have:
Playbook:
---
- hosts: localhost
gather_facts: false
vars:
instance_filter:
tag:environment: qa
tag:instance_id: i-123456789
tasks:
- name: print var
debug:
var: instance_filter
- name: manipulate the var
debug:
msg: "{{ instance_filter | dict2items | rejectattr('key', 'equalto', 'tag:instance_id') | list | items2dict }}"
Output:
PLAY [localhost] *******************************************************************************************************************************************************************************************************
TASK [print var] *******************************************************************************************************************************************************************************************************
ok: [localhost] => {
"instance_filter": {
"tag:environment": "qa",
"tag:instance_id": "i-123456789"
}
}
TASK [manipulate the var] **********************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": {
"tag:environment": "qa"
}
}
PLAY RECAP *************************************************************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
hope it helps.
Q: "How do you remove a JSON key?"
A: It's possible to create custom filter plugins. For example
$ cat filter_plugins/dict_utils.py
def dict_del_key(d, key):
del d[key]
return d
class FilterModule(object):
''' Ansible filters. Interface to Python dictionary methods.'''
def filters(self):
return {
'dict_del_key' : dict_del_key
}
The play below
- hosts: localhost
vars:
dict:
'tag:environment': 'qa'
'tag:instance_id': 'i-123456789'
tasks:
- debug:
msg: "{{ dict|dict_del_key('tag:instance_id') }}"
gives
msg:
tag:environment: qa
Notes:
See If you quote those config keys, they will become strings.
See the difference between 7.3.1. Double-Quoted Style and 7.3.2. Single-Quoted Style.
FWIW. See other filters available at GitHub.
I'm currently using Ansible 1.7.2. I have the following test playbook:
---
- hosts: localhost
tasks:
- name: set fact 1
set_fact: foo="[ 'zero' ]"
- name: set fact 2
set_fact: foo="{{ foo }} + [ 'one' ]"
- name: set fact 3
set_fact: foo="{{ foo }} + [ 'two', 'three' ]"
- name: set fact 4
set_fact: foo="{{ foo }} + [ '{{ item }}' ]"
with_items:
- four
- five
- six
- debug: var=foo
The first task sets a fact that's a list with one item in it. The subsequent tasks append to that list with more values. The first three tasks work as expected, but the last one doesn't. Here's the output when I run this:
PLAY [localhost] **************************************************************
GATHERING FACTS ***************************************************************
ok: [localhost]
TASK: [set fact 1] ************************************************************
ok: [localhost]
TASK: [set fact 2] ************************************************************
ok: [localhost]
TASK: [set fact 3] ************************************************************
ok: [localhost]
TASK: [set fact 4] ************************************************************
ok: [localhost] => (item=four)
ok: [localhost] => (item=five)
ok: [localhost] => (item=six)
TASK: [debug var=foo] *********************************************************
ok: [localhost] => {
"foo": [
"zero",
"one",
"two",
"three",
"six"
]
}
PLAY RECAP ********************************************************************
localhost : ok=6 changed=0 unreachable=0 failed=0
Given the with_items in task 4 and the fact that the output shows the task properly iterated over the items in that list, I would have expected the result to contain all the numbers zero through six. But that last task seems to only be evaluating set_fact with the last item in the list. Is this possibly a bug in Ansible?
Edit: I also just tested this on ansible 1.8 and the output was identical.
There is a workaround which may help. You may "register" results for each set_fact iteration and then map that results to list:
---
- hosts: localhost
tasks:
- name: set fact
set_fact: foo_item="{{ item }}"
with_items:
- four
- five
- six
register: foo_result
- name: make a list
set_fact: foo="{{ foo_result.results | map(attribute='ansible_facts.foo_item') | list }}"
- debug: var=foo
Output:
< TASK: debug var=foo >
---------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
ok: [localhost] => {
"var": {
"foo": [
"four",
"five",
"six"
]
}
}
As mentioned in other people's comments, the top solution given here was not working for me in Ansible 2.2, particularly when also using with_items.
It appears that OP's intended approach does work now with a slight change to the quoting of item.
- set_fact: something="{{ something + [ item ] }}"
with_items:
- one
- two
- three
And a longer example where I've handled the initial case of the list being undefined and added an optional when because that was also causing me grief:
- set_fact: something="{{ something|default([]) + [ item ] }}"
with_items:
- one
- two
- three
when: item.name in allowed_things.item_list
I was hunting around for an answer to this question. I found this helpful. The pattern wasn't apparent in the documentation for with_items.
https://github.com/ansible/ansible/issues/39389
- hosts: localhost
connection: local
gather_facts: no
tasks:
- name: set_fact
set_fact:
foo: "{{ foo }} + [ '{{ item }}' ]"
with_items:
- "one"
- "two"
- "three"
vars:
foo: []
- name: Print the var
debug:
var: foo
Jinja 2.6 does not have the map function. So an alternate way of doing this would be:
set_fact: foo="{% for i in bar_result.results %}{{ i.ansible_facts.foo_item }}{%endfor%}"
Below works for me:
- name: set fact
set_fact:
foo_item: "{{foo_item | default([]) + [item]}}"
loop:
- four
- five
- six
Updated 2018-06-08: My previous answer was a bit of hack so I have come back and looked at this again. This is a cleaner Jinja2 approach.
- name: Set fact 4
set_fact:
foo: "{% for i in foo_result.results %}{% do foo.append(i) %}{% endfor %}{{ foo }}"
I am adding this answer as current best answer for Ansible 2.2+ does not completely cover the original question. Thanks to Russ Huguley for your answer this got me headed in the right direction but it left me with a concatenated string not a list. This solution gets a list but becomes even more hacky. I hope this gets resolved in a cleaner manner.
- name: build foo_string
set_fact:
foo_string: "{% for i in foo_result.results %}{{ i.ansible_facts.foo_item }}{% if not loop.last %},{% endif %}{%endfor%}"
- name: set fact foo
set_fact:
foo: "{{ foo_string.split(',') }}"
Looks like this behavior is how Ansible currently works, although there is a lot of interest in fixing it to work as desired. There's currently a pull request with the desired functionality so hopefully this will get incorporated into Ansible eventually.