How to run JUnit tests on multiple Ansible assertions - junit

Question

I've got a GitLab pipeline that runs some Ansible playbooks in a container. At the end, I want to run some tests to ensure that everything is properly configured, e.g.
- name: Get the Frobzy value
raw: "frobzy -x foobar"
register: frobzy_val
changed_when: false
- name: Check that Frobzy is valid
assert:
that:
- frobzy_val.stdout is match "oogah"
fail_msg: "bad frobzy"
ignore_errors: true # see comments
- name: Get the Blobzy value
raw: "blobzy -y barfoo"
register: blobzy_val
changed_when: false
- name: Check that Blobzy is valid
assert:
that:
- blobzy_val.stdout is match "warra"
fail_msg: "bad blobzy"
ignore_errors: true # see comments
This is all nicely configured so that I get a JUnit report that gets displayed on my GitLab pipeline results.
The problem is, let's say both of these tests fail. If I look at the test results from my pipeline, it correctly displays that the tests failed. Yet the pipeline itself is marked as having passed! If I hadn't taken a detailed look at the test results, I would have assumed that everything is tickety-boo, and merged some potentially broken code.
The reason why the pipeline is marked as successful is because of those ignore_errors: true settings. It registers an error, but follows the directive that this should not fail the pipeline. But if I set ignore_errors: false (which is the default behavior), then as soon as the Frobzy test fails, the pipeline will abort and show a failure on the Frobzy test, but since it never ran the Blobzy test, I don't know that Blobzy would have failed, too.
Basically, I want a way to run all my Ansible assertions, even if some of them fail; I want to see all the test results in my pipeline, and I want the pipeline to fail if any tests failed.
How can I do this?

Answer 1

Here's a quick solution that might help.
Have a task to check if any of the tasks failed. Set a failed variable to true in that case.
Fail the playbook at the end if this variable is set to true.
- hosts: localhost
vars:
failed: false
gather_facts: no
tasks:
- name: Get the Frobzy value
raw: "echo frobzy"
register: frobzy_val
changed_when: false
- name: Check that Frobzy is valid
assert:
that:
- frobzy_val.stdout is match "frobzy"
fail_msg: "bad frobzy"
ignore_errors: true # see comments
- name: Get the Blobzy value
raw: "echo blobzy"
register: blobzy_val
changed_when: false
- name: Check that Blobzy is valid
assert:
that:
- blobzy_val.stdout is match "blobzy"
fail_msg: "bad blobzy"
ignore_errors: true # see comments
- name: Set a variable to true if the task fails
set_fact:
failed: true
ignore_errors: true
when: not ("'blobzy' in blobzy_val.stdout" ) or
not ("'frobzy' in frobzy_val.msg")
- name: Fail if failed is true
fail:
msg: "Failed"
when: failed
Run
─ ansible-playbook test.yaml
PLAY [localhost] ********************************************************************************************************************************************************************************
TASK [Get the Frobzy value] *********************************************************************************************************************************************************************
ok: [localhost]
TASK [Check that Frobzy is valid] ***************************************************************************************************************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [Get the Blobzy value] *********************************************************************************************************************************************************************
ok: [localhost]
TASK [Check that Blobzy is valid] ***************************************************************************************************************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [Set a variable to true if the task fails] *************************************************************************************************************************************************
skipping: [localhost]
TASK [Fail if failed is true] *******************************************************************************************************************************************************************
skipping: [localhost]
PLAY RECAP **************************************************************************************************************************************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0

Answer 2

Make a single assertion for both conditions and delay it until the end so that it fails after the two tests are done.
Add a debug task to show results for your pipelines.
- name: Get the Frobzy value
raw: "frobzy -x foobar"
register: frobzy_val
changed_when: false
- name: Get the Blobzy value
raw: "blobzy -y barfoo"
register: blobzy_val
changed_when: false
- name: Show results for Frobzy and Blobzy
debug:
msg:
- "Frobzy is {{ 'ok' if frobzy_val.stdout is match 'oogah' else 'bad' }}"
- "Blobzy is {{ 'ok' if lobzy_val.stdout is match 'warra' else 'bad' }}"
- name: Fail if any test is bad
assert:
that:
- frobzy_val.stdout is match "oogah"
- blobzy_val.stdout is match "warra"
fail_msg: "At least one of Frobzy or Blobzy has bad results. See debug above"
``