I have a package that is a core dependency of multiple other packages within my organization. My goal is to write an action to automate/ facilitate testing of these reverse dependencies. Roughly, the action should:
Trigger on a comment in a PR.
Run the unit tests of a set of reverse-dependencies with the code in that PR.
Reply to the PR with a comment about which tests failed (if any).
Steps 1 and 3 I got to work, but I’m running into issues with step 2. My current solution is to hardcode all job outputs to pass the results from step 2 to step 3, but I'm wondering if there is a way to avoid hardcoding this.
This following example workflow illustrates my problem:
name: Test
on: push
jobs:
unit-tests:
runs-on: ${{ matrix.os }}
continue-on-error: true
name: ${{ matrix.os }} (${{ matrix.pkg }})
strategy:
fail-fast: false
matrix:
# there will be more pkgs and OSes
os: [ubuntu-latest]
pkg: [pkgA, pkgB]
# how to avoid hardcoding these?
outputs:
ubuntu-latest-pkgA: ${{ steps.update-output.outputs.ubuntu-latest-pkgA }}
ubuntu-latest-pkgB: ${{ steps.update-output.outputs.ubuntu-latest-pkgB }}
steps:
- uses: actions/checkout#v2
- name: fake unit tests
run: |
exit 1 # fail all tests for now
shell: bash
- name: set error if tests fail
id: update-output
if: ${{ failure() }}
run: echo "::set-output name=${{ matrix.os }}-${{ matrix.pkg }}::error"
shell: bash
aggregate-results:
runs-on: ubuntu-latest
needs: unit-tests
steps:
- name: Aggregate results
env:
NEEDS: ${{ toJSON(needs) }}
run: echo "$NEEDS"
The job aggregate-results (inspired by this post) works nicely and prints:
{
"unit-tests": {
"result": "success",
"outputs": {
"ubuntu-latest-pkgA": "error",
"ubuntu-latest-pkgB": "error"
}
}
}
which I can use to create an informative comment. However, the job unit-tests requires me to hardcode the outputs for all combinations of os and pkg. Is there a way to do this dynamically?
I'm looking for this solution as well.
The only way to accomplish this approach is switch to re-usable workflows which allow to access one specific job context
https://docs.github.com/en/actions/learn-github-actions/contexts#jobs-context
Related
I am struggling with the recent change on how GitHub Action Workflows defines runtime variables, replacing the "set-ouput" approach with environment variables.
Last summer, it took me a couple of hours to figure out the code below working for my requirement. Define a matrix of OS and python versions, so that the CI workflow can create corresponding environments and run the pytests on it.
Is there any chance to get support on how to best transform to the new approach?
env:
# JSON variables (used in our strategy/matrix)
SUPPORTED_PYTHON_VERSIONS: '\"python-version\":[\"3.8\", \"3.9\"]'
SUPPORTED_OPERATING_SYSTEMS: '\"os\":[\"ubuntu-latest\", \"macos-latest\", \"windows-latest\"]'
jobs:
# The set-env job translates the json variables to a usable format for the workflow specifications.
set-env:
runs-on: ubuntu-latest
outputs:
py-os-matrix: ${{ steps.set-matrix-vars.outputs.py-os-matrix }}
# ^ this represents:
# matrix:
# - os: [ubuntu-latest, ...]
# - python-version: [3.7, ...]
os-matrix: ${{ steps.set-matrix-vars.outputs.os-matrix }}
# ^ this represents:
# matrix:
# - os: [ubuntu-latest, ...]
steps:
- id: set-matrix-vars
run: |
echo "::set-output name=py-os-matrix::{${{ env.SUPPORTED_PYTHON_VERSIONS }},${{ env.SUPPORTED_OPERATING_SYSTEMS }}}"
echo "::set-output name=os-matrix::{${{ env.SUPPORTED_OPERATING_SYSTEMS }}}"
test-run:
name: test on ${{ matrix.os }} - ${{ matrix.python-version }}
needs: set-env
strategy:
fail-fast: true
matrix: ${{ fromJson(needs.set-env.outputs.py-os-matrix) }}
Meanwhile, a more elegant way of using the matrix keyword functions is available. My new implementation is much cleaner and, most important, avoids the above-mentioned deprecated function of printing env variables to stdout. However, the point is open how to integrate python versions >=3.10 ...
jobs:
test-run:
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [3.8, 3.9]
defaults:
run:
shell: bash
runs-on: ${{ matrix.os }}
steps:
- name: Check out repository code
uses: actions/checkout#v3
- name: Install python
id: setup-python
uses: actions/setup-python#v4
with:
python-version: ${{ matrix.python-version }}
check-latest: true
Every further step which comes after the one with setup-python is executed in the currenlty selected combination of OS and python.
I want to use environment variables at the job level. Is there a way to do it?
env:
stageEnv: UAT
jobs:
name: Upload Build
if: ${{ env.stageEnv == 'UAT' }}
steps:
....
I get unrecognized named-value: 'env' error. Tried $stageEnv and ${{ env.stageEnv }}
Note: It works when I access within 'steps', but would like this to be accessible at 'jobs' level.
I'm afraid not, but you can do like this:
env:
stageEnv: UAT
jobs:
build:
name: Build
runs-on: ubuntu-latest
outputs:
stageEnv: ${{ steps.init.outputs.stageEnv }}
steps:
- name: Make environment variables global
id: init
run: |
echo "stageEnv=${{ env.stageEnv }}" >> $GITHUB_OUTPUT
And use it in another job:
upload:
name: Upload build
needs: build
if: ${{ needs.build.outputs.stageEnv == 'UAT' }}
Note this is just an example, and I personally prefer environment variables uppercase and output variables lowercase
We have two runners, one for running production jobs and another for running non production jobs, but I am unable to do that using a workflow level environment variable.
Below is what I have:
name: Workflow file
on:
workflow-dispatch
env:
RUNNER_NAME: ${{ contains(github.ref, 'main') && 'Prod Runner' || 'non-Prod Runner' }}
jobs:
job-run:
runs-on: [${{ env.RUNNER_NAME }}]
needs: ...
steps:
..........
I get the following error message:
Invalid workflow file
You have an error in your yaml syntax on line ###
How do I do this? I don't want to have separate workflow files for prod and non-prod workflows.
For what you can check on this Github Actions ISSUE, it seems it's not possible to use natively env variable on the runs-on job field (yet?).
However, there is a workaround if you configure a variable as output in a previous job, so you would be able to use it afterwards.
Example: runs-on: ${{ needs.setup.outputs.runner }}
In your case, the workflow would look like this:
on:
workflow_dispatch:
jobs:
setup:
runs-on: ubuntu-latest
outputs:
runner: ${{ steps.step1.outputs.runner }}
steps:
- name: Check branch
id: step1
run: |
if [ ${{ github.ref }} == 'refs/heads/main' ]; then
echo "::set-output name=runner::ubuntu-latest"
else
echo "::set-output name=runner::macos-latest"
fi
job1:
needs: [setup]
runs-on: ${{ needs.setup.outputs.runner }}
steps:
- run: echo "My runner is ${{ needs.setup.outputs.runner }}" #ubuntu-latest if main branch
I've made a test here if you want to have a look:
workflow file
workflow run
I'm trying to make a GitHub Actions workflow where one job has a dynamic value to its environment setting. https://docs.github.com/en/actions/reference/environments I have two jobs in this workflow. The first job determines which environment the second job will run in, and this is in turn based on which git branch the Actions job is run from.
This is my naive attempt to make it work, but I get an error which says
(Line: 29, Col: 18): Unrecognized named-value: 'needs'. Located at position 1 within expression: needs.get-environment.outputs.environment_name
name: Environments
on:
push:
workflow_dispatch:
jobs:
get-environment:
runs-on: ubuntu-latest
outputs:
environment_name: ${{ steps.get_environment.outputs.environment_name }}
steps:
- id: get_environment
run: |
if [ "$GITHUB_REF" = "refs/heads/test" ]
then
echo "::set-output name=environment_name::test"
elif [ "$GITHUB_REF" = "refs/heads/qa" ]
then
echo "::set-output name=environment_name::qa"
elif [ "$GITHUB_REF" = "refs/heads/master" ]
then
echo "::set-output name=environment_name::production"
fi
use-environment:
runs-on: ubuntu-latest
needs: [get-environment]
environment: ${{ needs.get-environment.outputs.environment_name }}
steps:
- uses: actions/checkout#v2
- name: Run a one-line script
run: echo ${{ secrets.ENV_DEPENDENT_SECRET }}
Is it possible to achieve what I'm trying to do? My end goal is to have a single workflow file for three different app environments (test, QA and prod), where each app environment uses a separate Actions environment. (terminology gets confusing, I know)
I actually had to solve this issue for the place I work. You can use a matrix with the list of your env names and dynamically set the environment name. Take a look at the link for the workflow file.
Have you tried
> use-environment:
> runs-on: ubuntu-latest
> needs: [get-environment]
> environment:
> name: ${{ needs.get-environment.outputs.environment_name }}
I am trying to execute some option steps if the previous step is failing.
I need to download the old artifacts in order to avoid that my Terraform build action is not detecting changes. Therefore I added a diff action to identify if the docker file to build the zip-layer has changed, if there is no change the old artifacts from a previous execution should be downloaded. In some cases the last execution does not contain the artifacts e.g. failure of the jobs. In that case I would like to get the latest version based on the existing docker image.
Note: The code is part of a matrix execution but for simplicity I reduced the action to the problem area.
job_prepare:
.....
job_layers:
needs: job_prepare
runs-on: ubuntu-latest
strategy:
matrix: ${{fromJson(needs.job_prepare.outputs.layer_matrix)}}
steps:
- name: Checkout
uses: actions/checkout#v2
- name: matrix name
run: |
echo $GITHUB_WORKSPACE
echo ${{ matrix.path }}
- uses: technote-space/get-diff-action#v3
id: git_diff
with:
PREFIX_FILTER: ${{ matrix.prefix }}
SUFFIX_FILTER: Dockerfile
- name: LayerDockerBuild
id: layerDocker
if: steps.git_diff.outputs.diff
run: |
docker build ...
docker push ...
- name: Layer via Artifacts
if: (steps.git_diff.outputs.diff == false)
uses: dawidd6/action-download-artifact#v2
with:
workflow: review.yml
name: ${{ matrix.name }}
- name: Layer via Docker
if: steps.git_diff.outputs.diff && ${{ failure() }}
id: layerD
run: |
....
docker pull "docker.pkg.github.com/$REPO_NAME/$IMAGE_ID:$VERSION"
docker run --rm -v $GITHUB_WORKSPACE:/data docker.pkg.github.com/$REPO_NAME/$IMAGE_ID:$VERSION cp /packages/${{ matrix.name }}.zip /data
- name: Upload layer zip
if: ${{ always() }}
uses: actions/upload-artifact#v2
with:
name: ${{ matrix.name }}
path: ${{ matrix.name }}.zip
The problem is basically the logic in the line if: steps.git_diff.outputs.diff && ${{ failure() }}
thanks for your help on any hints how to make the step option when the diff is false and the step ~Layer via Artifacts~ is not failing.
To execute your step if the previous one was not a failure you can just set failure() without brackets. With the if, no need of brackets.
if: steps.git_diff.outputs.diff && failure()
Hope it will help