Chain testing and releasing workflows in GitHub Actions - github-actions

I have 2 conditions under which I want to run my workflows in GH actions on a go project
run unit tests on every PR (I have already taken care of this with the following workflow.yaml
name: unit-tests
on: [ pull_request ]
jobs:
unit-tests:
runs-on: Linux
steps:
- name: checkout project
uses: actions/checkout#v2
- name: dynamically retrieve go version
uses: arnested/go-version-action#v1
id: go-version
- name: setup go ${{ steps.go-version.outputs.minimal }}
uses: actions/setup-go#v2
with:
go-version: ${{ steps.go-version.outputs.minimal }}
- name: run unit tests
run: go test -race -cover -v ./... -run Unit
My second requirement is the following:
run unit tests AND proceed to creating a tag + release if the tests succeed and the target branch is master
My question is the following: Since the on clause is unique per workflow, is there a way I can leverage the above workflow (without the need to write a more or less clone) that can be integrated in the second use case?

First, reusing workflows is available since Oct. 2021
That means you can make a workflow on master branch, which would "call" (reuse) your second (existing) workflow, itself with its on: directive
The call would look like:
jobs:
call-workflow-passing-data:
uses: octo-org/example-repo/.github/workflows/reusable-workflow.yml#main
with:
username: mona
secrets:
envPAT: ${{ secrets.envPAT }}

Related

Can GitHub Action Job continue from the state the previous job left (without artifact or complicated procedure)?

I previously had a GitHub Action with a single Job that was doing 3 things:
Build the .net application
Run Unit Tests
Run Integration Tests
Now, I splitted this job in 3 different ones because:
I like to experiment
I like to see the GitHub PR updating the steps separately
I can/want run the Unit and Integration tests in parallel so the entire process can complete quickly
This is the current GitHub Action:
name: Pull Request Checks
on:
pull_request:
types: [opened, synchronize, reopened, labeled]
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Buid VS solution
id: build
run: dotnet build "FSharp project/MyProject.sln" -c RELEASE
unit-tests:
name: Unit Tests
needs: [build]
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Unit Tests
id: unit-tests
run: dotnet test "FSharp project/UnitTests/UnitTests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY"
integration-tests:
name: Integration Tests
needs: [build]
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Integration Tests
id: integration-tests
if: github.event.action == 'labeled' && github.event.label.name == 'pr:ready'
run: dotnet test "FSharp project/IntegrationTests/IntegrationTests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY"
Ideally the Integration Tests job run only when the PR is labeled "pr:ready" (this point has still to be tuned/solved maybe).
This entire process works.
I had to duplicate the Checkout step in each job, this means they are completely different "machine".
If that is true, why the dotnet test with --no-build is still able to work?
MS changed the behaviour of that flag, so honestly I don't remember if the dotnet cli version is running here is able to reuse the possibly executed build or it runs a build if needed itself.
So I'm not entirely sure the Checkout results to have a completely fresh environment in "successive" jobs, and if this is the case... there is a way to reuse the previous "state" in a simple way (like a simple parameter, not using artifacts and similar stuff)?
Every GitHub actions job is run on a fresh virtual environment. The only way to get data/files/etc between jobs is artifacts. In your case, the solution would be to generate an artifact in the build job which can then be used in the two testing jobs in parallel. See this issue on the checkout action repo asking the same thing.
So you could wrap up everything you need for the tests into an artifact and replace the checkout step in the test jobs with downloading that artifact.
As to why your --no-build tests work, that is hard to answer without knowing what exactly is being checked out.

How to make a job that doesn't depend on a matrix only run once

I have a GitHub Actions workflow containing a setup job:
setup:
runs-on: windows-latest
steps:
- name: Checkout Repository
uses: actions/checkout#v2.4.0
- name: Install .NET
uses: actions/setup-dotnet#v1.8.2
with:
dotnet-version: 6.0.x
and once setup is complete, I build for my for target platforms:
build:
needs:
- setup
runs-on: windows-latest
strategy:
matrix:
target:
- win-x64
- linux-x64
- linux-arm64
- osx-x64
self-contained:
- self-contained
- framework-dependent
steps:
- name: Checkout Repository
uses: actions/checkout#v2.4.0
- name: Install .NET
uses: actions/setup-dotnet#v1.8.2
with:
dotnet-version: 6.0.x
- name: Build
run: >-
dotnet publish -r ${{matrix.target}} --self-contained ${{
matrix.self-contained == 'self-contained' }} ${{ matrix.self-contained
&& '' || '/p:DisablePatch="--nopatch"' }}
In build, I end up duplicating the work done by setup—8 times. Is there a way to avoid this, and perhaps just copy the machine state from setup into the build job?
When you run an action (in your job steps), it will only apply to the runner used to execute the job steps.
According to the Github documentation about matrix, each matrix job runs in parallel. Therefore they will all need the setup to be executed as each provided runner starts without the Checkout Repository and Install .NET steps configured.
You can't share those setup between the runners provided by Github, because each job will use a new (and different) runner.
Which means your setup job here won't do anything that requires the needs configuration on the build job, as what it does will only applies to itself.
Therefore, removing the needs: setup on the build job would be the same as what you did.
What you could do instead for example, is using self-hosted runners with dotnet already installed, or a docker image. In that case you wouldn't need to setup dotnet every time, but is it worth the cost? (as building a docker image can take more time that doing the setup, so you should evaluate this first).

Only run GitHub Actions step if not a pull request

I have a workflow which needs to execute either on a push or a pull request with the exception of the last step which pushes a package to NuGet (I don't want this to occur on a pull request, even to master).
How can I prevent the Publish NuGet step from running if the workflow is triggered from a pull request?
name: .NET Core
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Setup .NET Core
uses: actions/setup-dotnet#v1
with:
dotnet-version: 3.1.101
- name: Install dependencies
run: dotnet restore
- name: Build
run: dotnet build --configuration Release --no-restore
- name: Test
run: dotnet test --no-restore --verbosity normal
- name: Publish NuGet
uses: brandedoutcast/publish-nuget#v2.5.2
with:
PROJECT_FILE_PATH: "Orleans.Sagas/Orleans.Sagas.csproj"
NUGET_KEY: ${{secrets.NUGET_KEY}}
You can inspect the github.event_name context variable which contains the name of the event that triggered the workflow run. (eg, pull_request or push)
In this case, you can run a step for all events whose name is not pull_request with a github.event_name != 'pull_request' conditional on your step.
For example:
- name: Publish NuGet
uses: brandedoutcast/publish-nuget#v2.5.2
with:
PROJECT_FILE_PATH: "Orleans.Sagas/Orleans.Sagas.csproj"
NUGET_KEY: ${{secrets.NUGET_KEY}}
if: github.event_name != 'pull_request'
For future travellers, I found this action that worked quite well. I just needed to do an if: needs.pr-check.outputs.number != 'null' in order to filter by things being a PR or not.
https://github.com/8BitJonny/gh-get-current-pr
github.event_name != 'pull_request' did not work for me because the on.pull_request trigger doesn't exist for workflows that aren't launched by a PR.

In a github actions workflow, is there a way to have multiple jobs reuse the same setup?

I recently hooked up my project with github actions for continuous integration. I created two separate jobs: the first one checks if the code in the pull request is accepted by our linter, and the second one checks if the code passes the test suite. I like that having two jobs like this shows up as two separate checkmarks in the Github webpage for the pull request:
The problem I'm having now is that there is some duplicated code in workflow YAML file: the first 3 steps, which install Lua and Luarocks. Not only is it annoying to maintain, but it also wastes CI minutes by running the same actions twice. Is there a way to avoid this? So that the setup code is only written in one place, and only runs once when the workflow executes?
But I am confused what would be the proper way to proceed:
Should I create my own Github Action with the shared setup code?
Should I create a Docker image that already has Lua and Luarocks pre-installed?
Should I use a single job? Can I still have independent checkmarks for the linter and the test suite if they are steps of the same job?
Something else?
Here is the current YAML file for my workflow:
name: Github Actions CI
on: [ pull_request ]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: leafo/gh-actions-lua#v8.0.0
- uses: leafo/gh-actions-luarocks#v4.0.0
- run: luarocks install luacheck
- run: ./run-linter.sh
test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: leafo/gh-actions-lua#v8.0.0
- uses: leafo/gh-actions-luarocks#v4.0.0
- run: luarocks install busted
- run: ./build-project.sh
- run: ./run-test-suite.sh
I tried searching for similar questions but couldn't find anything that exactly answered my question:
Caching APT packages in GitHub Actions workflow: I can't use this solution because I don't have a way to precisely specify all the versions of all the dependencies that I am using, so that they may be cached. I also don't mind if separate runs of the workflow are not cached. I'm more worried about the code duplication.
Github actions share workspace/artifacts between jobs? I don't want to have to manage uploading uploading artifacts to a separate service and then deleting them afterwards.
Reuse portion of github action across jobs: In that question the only difference between the jobs is a single variable, so accepted answer is to use a build matrix. But I don't think a build matrix would work as well in my case, where only the setup code is the same?
As of today (August 2021) composite action is no longer limited to run. GitHub Actions: Reduce duplication with action composition
name: "Publish to Docker"
description: "Pushes built artifacts to Docker"
inputs:
registry_username:
description: “Username for image registry”
required: true
registry_password:
description: “Password for image registry”
required: true
runs:
using: "composite"
steps:
- uses: docker/setup-buildx-action#v1
- uses: docker/login-action#v1
with:
username: ${{inputs.registry_username}}
password: ${{inputs.registry_password}}
- uses: docker/build-push-action#v2
with:
context: .
push: true
tags: user/app:latest
Old Answer
What you are looking for is composite action which help you reuse once defined set of steps.
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: octocat/say-hello#v1
- run: luarocks install luacheck
- run: ./run-linter.sh
test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: octocat/say-hello#v1
- run: luarocks install busted
- run: ./build-project.sh
- run: ./run-test-suite.sh
octocat/say-hello/action.yml:
runs:
using: "composite"
steps:
- run: echo "Nice to meet you!"
shell: pwsh
For more details you can also check's ADR here.
And why you can't simply run it once for all jobs, because each job may run on a different machine.
A job is a set of steps that execute on the same runner. By default, a workflow with multiple jobs will run those jobs in parallel. You can also configure a workflow to run jobs sequentially. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.
There are 3 main approaches for code reusing in GitHub Actions:
Reusable Workflows
Dispatched workflows
Composite Actions
There is an article describing their pros and cons.
In your case if the duplicated steps are in the single workflow you also can:
extract them to the "preparation" job
upload build artifacts
add "preparation" job to "needs" key of both jobs
download build artifact in both jobs
The below code snippet should do the work for you. Instead of creating two jobs, one for lint and another for test, you can use one job and create multiple tasks
name: Github Actions CI
on: [ pull_request ]
jobs:
lint:
name: Lint and Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: leafo/gh-actions-lua#v8.0.0
- uses: leafo/gh-actions-luarocks#v4.0.0
- name: Install Luacheck
run: luarocks install luacheck
- name: Run Lint Check
- run: ./run-linter.sh
- name: Install Busted
run: luarocks install busted
- name: Build Project
run: ./build-project.sh
- name: Run Test Suite
- run: ./run-test-suite.sh

Does Github Actions have templates

As I have repetitve steps in my Github Actions, I would like to create a template. Let's make a example
name: ci
on: ["push"]
jobs:
build-and-test:
strategy:
matrix:
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- name: checkout
uses: actions/checkout#v1
- name: do stuff
run: |
bash stuff
Is it possible to save only the steps in a separated file? And import afterwards?
Unfortunately it does not look like github-actions supports reusing workflows. Not even YAML anchors are supported.
It looks like the only way to share steps (not setup) is to create actions.
Update: A storm brewing
I have also caught wind of the possibility of reusing actions. Follow the issue to stay up-to-date.
I mentioned in "Reuse portion of GitHub action across jobs" that reusing GitHub Worfflow is now (Oct. 2021) available.
The documentation "Reusing workflows" includes a section "Reusable workflows and workflow templates", which leads to "Creating workflow templates"
If you need to refer to a repository's default branch, you can use the $default-branch placeholder.
When a workflow is created using your template, the placeholder will be automatically replaced with the name of the repository's default branch.
For example, this file named octo-organization-ci.yml demonstrates a basic workflow.
name: Octo Organization CI
on:
push:
branches: [ $default-branch ]
pull_request:
branches: [ $default-branch ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Run a one-line script
run: echo Hello from Octo Organization