With the matrix strategy and two jobs like those below, the first job runs a "matrix" of possible configurations in parallel, and when all of them succeed, it moves onto the second job.
jobs:
job1:
runs-on: ubuntu-latest
strategy:
matrix:
version: [10, 12, 14]
steps:
# etc
job2:
runs-on: ubuntu-latest
needs: job1
strategy:
matrix:
version: [10, 12, 14]
steps:
# etc
What would be nice is to essentially have a matrix over the whole workflow, where for each matrix value, the entire workflow of jobs is run unimpeded, in parallel. That means for a particular configuration job2 can proceed right after job1 completes for that configuration, regardless of whether job1 for another configuration is still in process or has failed.
In the following diagram, the first case is what will currently happen, and the second is the desired case:
I feel like this could be "emulated" by launching the workflow several times with a different context. But how could this be done, for example from a trigger? Somehow specify on: push that 3 versions of the workflow must run simultaneously?
After some digging, I think "Reusable workflows" is the currently supported method.
The solution is to have two workflow files. One that has all the jobs, and another small workflow that has the matrix and calls the other workflow.
name: Common Workflow
on:
workflow_call:
inputs:
version: # the variable you can use in place of a matrix
required: true
type: number
jobs:
job1:
runs-on: ubuntu-latest
steps:
- run: echo 'job1 version ${{ inputs.version }}'
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo 'job2 version ${{ inputs.version }}'
name: Master Workflow
on:
push:
jobs:
version-matrix:
strategy:
# super important if you want to see all results, even if one fails
# fail-fast is true by default
fail-fast: false
matrix:
version: [10, 12, 14]
uses: ./.github/workflows/common-workflow.yml # calls the one above ^
with:
version: ${{ matrix.version }}
secrets: inherit
Here's a real example of it running in the UI:
One thing you will run into once you run this though, is the UI lists the subjobs in alphabetical order, and not the order they are listed in the workflow. So you lose the sense of the dependencies between the jobs. This is true in February 2023, so maybe Github will improve the UI down the road. Until then, we got around this by naming our jobs like 1. First Job, 2. Second Job so alphabetical is equivalent to the order they run in. It's hacky but it works.
Another thing to note, especially for deployments is the fail-fast attribute. This should be set to false to make sure every workflow in the matrix gets to run to completion, even if one of them fails.
Related
Is there a way to use a higher resource runner machine for a specific value in matrix strategy?
this is a sample GitHub action:
test-app:
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
matrix:
run: ['app1', 'app2', 'app3']
for app3 I need 2x resources on the runner machine. Could someone please help me? I searched a lot but could find the answer here on stackoverflow
You need two changes to make this happen:
Define an object to reference your runs-on values
Reference those in the job definition
I have only been able to do this by referencing outputs from a previous job:
jobs:
test-setup:
runs-on: ubuntu-latest
outputs:
runners: '{"app1":"ubuntu-latest", "app2": "ubuntu-latest", "app3": "ubuntu-22.04-16core"}'
steps:
- run: echo no-op
test-app:
needs: [test-setup]
runs-on: ${{ fromJSON(needs.test-setup.outputs.runners)[matrix.run] }}
timeout-minutes: 60
strategy:
matrix:
run: ['app1', 'app2', 'app3']
steps:
- run: echo ${{ matrix.run }}
This is the runs-on value: ${{ fromJSON(needs.test-setup.outputs.runners)[matrix.run] }}, it just pulls the string from the previous job's output
I am trying to run a job on multiple os using matrix strategy. my code is below -
strategy:
matrix:
os: [ubuntu-18.04, ubuntu-20.04, self-hosted]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout#v2
However if one of the triggerd job complete with success or failure, all other paralell jobs on remaining os from matrix get terminated automatically.
How can I change this behavior?
I am expecting my all jobs from matrix to keep running till end.
Thanks to answer here - https://stackoverflow.com/a/70841204/4042883
adding below attribute helped -
strategy:
fail-fast: false
When running a GitHub Actions matrix workflow, how can we allow a job to fail, continue running all the other jobs, and also mark the workflow itself as failed?
Below in this image, you can see that the workflow passes even after a job failed. We need to mark the workflow as failed in this case.
Here is the small portion of my workflow yaml file.
continue-on-error line will continue the workflow even if a job fails but how do we get the whole workflow marked as failed?
matrixed:
runs-on: ubuntu-latest
continue-on-error: true
timeout-minutes: 60
defaults:
run:
shell: bash
working-directory: myDir
strategy:
matrix:
testgroups:
[
"bookingpage-docker-hub-parallel",
"bookingpage-docker-hub-parallel-group-1",
"bookingpage-payments",
]
I did find this unanswered question, but this is about steps and we need to know about jobs.
Use fail-fast: false for the strategy and don't set continue-on-error on the job.
matrixed:
runs-on: ubuntu-latest
timeout-minutes: 60
defaults:
run:
shell: bash
working-directory: myDir
strategy:
fail-fast: false
matrix:
testgroups:
[
"bookingpage-docker-hub-parallel",
"bookingpage-docker-hub-parallel-group-1",
"bookingpage-payments",
]
I'm trying to achieve a flow with a job dependency tree based on needs and if
This is the workflow file, and my problem is that the job4 is not being triggered even when the job3 is being executed successfully.
I can't find any documentation explaining what is happening, but it seems that the needs tree that is being built behind the scenes is not matching the job4, because the needs on the job3 is not completed (is forced by the if)
I have a workaround to solve this flow using job outputs, but I was wondering why the simple example is not working. Can anyone explain me why?
Thanks
name: Test Workflow
on:
push:
jobs:
job1:
runs-on: ubuntu-latest
steps:
- run: echo Job1
job2:
if: cancelled()
runs-on: ubuntu-latest
steps:
- run: echo Job2
job3:
if: always()
needs: [job1, job2]
runs-on: ubuntu-latest
steps:
- run: echo Job3
job4:
needs: job3
runs-on: ubuntu-latest
steps:
- run: echo Job4
All dependent jobs must succeed. So jon your job 4 depdends on the job 3 it must that job 3 and all it's need must be succeded.
Please check this GitHub issue - Job-level "if" condition not evaluated correctly if job in "needs" property is skipped.
I assume that in your case even if you forced job 3 to run the result is not success but something different. Thus job 4 is not run.
You probably need sth like this on job 4.
if: |
always() &&
(needs.job3.result == 'success' || needs.job3.result == 'skipped')
I would like to allow a job to run only if its name can be found in some file.
jobs:
job1:
if : run some script that returns true if "${{ github.job }}" is in file.txt
steps:
...
However I'm quite sure i cant run a script outside of run, which I believe can't appear before the steps section, right?
If the option above simply isn't possible, perhaps there is a way to run this script on the first step and have the next steps not run unless the said critical step passed?
Thanks!
You could for example use 2 jobs in your action, where the second job would only be executed if the first one succeed return true using an if condition and outputs.
The workflow would look like something as below:
jobs:
job1:
runs-on: ubuntu-latest
outputs:
output1: ${{ steps.script.outputs.bool }}
steps:
- name: Run the python script
id: script
run: |
value=$((your command running the script here) 2> &1) # should return true
echo "::set-output name=bool::$value" # set the value as output
job2:
runs-on: ubuntu-latest
# Wait from the job1 to be completed before starting job2
needs: job1
if ${{ ${{needs.job1.outputs.output1}} == true }} # won't execute if output of job 1 isn't true
steps:
...
If you want to know more about:
if condition + context and expression
outputs
Note: The contains expression in the if condition could also be used eventually depending on your context (what the script does). It could even be the easiest solution :)