I just start with GitHub Actions and I'm trying to configure correctly jobs. Now I have a job - build which set up python and installs dependencies, I have a job with behave test too which needs the dependencies to run.
When I have the test and build in the one job, everything works fine. But I want to have build and test in separate jobs. But when I run them in this configuration, I get the error behave: command not found. I install the Behave in requirementx.txt file. What am I doing wrong? Is this configuration generally possible?
name: CI test
on:
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up Python 3.8
uses: actions/setup-python#v2
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
cc_test:
needs: build
runs-on: ubuntu-latest
steps:
- name: Run cc test
run: |
behave --no-capture --no-skipped -t guest -t cc -D driver=BROWSERSTACK features
As riQQ and documentation says
A job is a set of steps that execute on the same runner. By default, a workflow with multiple jobs will run those jobs in parallel. You can also configure a workflow to run jobs sequentially. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.
In your case it would be the best to have one job build and test and do both things in this one job. Putting tests in separate jobs can be a good move, but it would require one of two:
prepare testable package in previous step and share it (it could still requires to install some dependencies)
checkout code, install dependencies, build code and run tests what means that you need to repeat all steps from previous job
Related
I previously had a GitHub Action with a single Job that was doing 3 things:
Build the .net application
Run Unit Tests
Run Integration Tests
Now, I splitted this job in 3 different ones because:
I like to experiment
I like to see the GitHub PR updating the steps separately
I can/want run the Unit and Integration tests in parallel so the entire process can complete quickly
This is the current GitHub Action:
name: Pull Request Checks
on:
pull_request:
types: [opened, synchronize, reopened, labeled]
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Buid VS solution
id: build
run: dotnet build "FSharp project/MyProject.sln" -c RELEASE
unit-tests:
name: Unit Tests
needs: [build]
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Unit Tests
id: unit-tests
run: dotnet test "FSharp project/UnitTests/UnitTests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY"
integration-tests:
name: Integration Tests
needs: [build]
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Integration Tests
id: integration-tests
if: github.event.action == 'labeled' && github.event.label.name == 'pr:ready'
run: dotnet test "FSharp project/IntegrationTests/IntegrationTests.fsproj" -c Release --no-build --filter "TestCategory!=SKIP_ON_DEPLOY"
Ideally the Integration Tests job run only when the PR is labeled "pr:ready" (this point has still to be tuned/solved maybe).
This entire process works.
I had to duplicate the Checkout step in each job, this means they are completely different "machine".
If that is true, why the dotnet test with --no-build is still able to work?
MS changed the behaviour of that flag, so honestly I don't remember if the dotnet cli version is running here is able to reuse the possibly executed build or it runs a build if needed itself.
So I'm not entirely sure the Checkout results to have a completely fresh environment in "successive" jobs, and if this is the case... there is a way to reuse the previous "state" in a simple way (like a simple parameter, not using artifacts and similar stuff)?
Every GitHub actions job is run on a fresh virtual environment. The only way to get data/files/etc between jobs is artifacts. In your case, the solution would be to generate an artifact in the build job which can then be used in the two testing jobs in parallel. See this issue on the checkout action repo asking the same thing.
So you could wrap up everything you need for the tests into an artifact and replace the checkout step in the test jobs with downloading that artifact.
As to why your --no-build tests work, that is hard to answer without knowing what exactly is being checked out.
I have a GitHub Actions workflow containing a setup job:
setup:
runs-on: windows-latest
steps:
- name: Checkout Repository
uses: actions/checkout#v2.4.0
- name: Install .NET
uses: actions/setup-dotnet#v1.8.2
with:
dotnet-version: 6.0.x
and once setup is complete, I build for my for target platforms:
build:
needs:
- setup
runs-on: windows-latest
strategy:
matrix:
target:
- win-x64
- linux-x64
- linux-arm64
- osx-x64
self-contained:
- self-contained
- framework-dependent
steps:
- name: Checkout Repository
uses: actions/checkout#v2.4.0
- name: Install .NET
uses: actions/setup-dotnet#v1.8.2
with:
dotnet-version: 6.0.x
- name: Build
run: >-
dotnet publish -r ${{matrix.target}} --self-contained ${{
matrix.self-contained == 'self-contained' }} ${{ matrix.self-contained
&& '' || '/p:DisablePatch="--nopatch"' }}
In build, I end up duplicating the work done by setup—8 times. Is there a way to avoid this, and perhaps just copy the machine state from setup into the build job?
When you run an action (in your job steps), it will only apply to the runner used to execute the job steps.
According to the Github documentation about matrix, each matrix job runs in parallel. Therefore they will all need the setup to be executed as each provided runner starts without the Checkout Repository and Install .NET steps configured.
You can't share those setup between the runners provided by Github, because each job will use a new (and different) runner.
Which means your setup job here won't do anything that requires the needs configuration on the build job, as what it does will only applies to itself.
Therefore, removing the needs: setup on the build job would be the same as what you did.
What you could do instead for example, is using self-hosted runners with dotnet already installed, or a docker image. In that case you wouldn't need to setup dotnet every time, but is it worth the cost? (as building a docker image can take more time that doing the setup, so you should evaluate this first).
I'm trying to build a repository which allows me to build Docker images for different versions of a project I'm forking.
The repo should have the following layout:
main branch where the workflow is defined, with a trigger such as:
on:
push:
branches-ignore:
- main
The workflow builds the software from any branch (basically a mvn clean package, docker build and docker push that applies to all versions of the software)
many software-1.2.3 branches which don't contain any .github/workflow files (it would be cumbersome to copy this file into each branch, and maintain it there)
From my research so far, it seems that GitHub Actions only runs when a workflow definition is present. However, I wonder if there's a way using webhooks or something to trick the system into doing what I want.
My next best option would probably be using workflow_dispatch,
on: push !main wouldn't work, because that !main branch would need to have this workflow in it.
Running the workflow manually is the easiest solution to implement.
on:
workflow_dispatch:
inputs:
BRANCH_OF_FORK:
description: 'branch name, on which this pipeline should run'
required: true
jobs:
doSomething:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
ref: ${{ github.event.inputs.BRANCH_OF_FORK }}
- run: docker build something something
There's a on:fork event, that could be used to trigger a run, but it could only trigger a fork event of this specific branch, that has the on:fork code in it's workflow.
It's possible to run the workflow on a cron job without on-fork event, but you would have to pick the correct branch programmatically.
steps:
- id: branch-selector
run: |
git clone ${{ github.event.repo.url }} .
all_branches=`git branch --all | grep -v " main\$" | grep -v "/main\$"`
correct_branch=`pick_correct_branch.py $all_branches`
git checkout -b $correct_branch
- run: docker build something something
pick_correct_branch.py is left as an exercise for the reader to implement.
I recently hooked up my project with github actions for continuous integration. I created two separate jobs: the first one checks if the code in the pull request is accepted by our linter, and the second one checks if the code passes the test suite. I like that having two jobs like this shows up as two separate checkmarks in the Github webpage for the pull request:
The problem I'm having now is that there is some duplicated code in workflow YAML file: the first 3 steps, which install Lua and Luarocks. Not only is it annoying to maintain, but it also wastes CI minutes by running the same actions twice. Is there a way to avoid this? So that the setup code is only written in one place, and only runs once when the workflow executes?
But I am confused what would be the proper way to proceed:
Should I create my own Github Action with the shared setup code?
Should I create a Docker image that already has Lua and Luarocks pre-installed?
Should I use a single job? Can I still have independent checkmarks for the linter and the test suite if they are steps of the same job?
Something else?
Here is the current YAML file for my workflow:
name: Github Actions CI
on: [ pull_request ]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: leafo/gh-actions-lua#v8.0.0
- uses: leafo/gh-actions-luarocks#v4.0.0
- run: luarocks install luacheck
- run: ./run-linter.sh
test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: leafo/gh-actions-lua#v8.0.0
- uses: leafo/gh-actions-luarocks#v4.0.0
- run: luarocks install busted
- run: ./build-project.sh
- run: ./run-test-suite.sh
I tried searching for similar questions but couldn't find anything that exactly answered my question:
Caching APT packages in GitHub Actions workflow: I can't use this solution because I don't have a way to precisely specify all the versions of all the dependencies that I am using, so that they may be cached. I also don't mind if separate runs of the workflow are not cached. I'm more worried about the code duplication.
Github actions share workspace/artifacts between jobs? I don't want to have to manage uploading uploading artifacts to a separate service and then deleting them afterwards.
Reuse portion of github action across jobs: In that question the only difference between the jobs is a single variable, so accepted answer is to use a build matrix. But I don't think a build matrix would work as well in my case, where only the setup code is the same?
As of today (August 2021) composite action is no longer limited to run. GitHub Actions: Reduce duplication with action composition
name: "Publish to Docker"
description: "Pushes built artifacts to Docker"
inputs:
registry_username:
description: “Username for image registry”
required: true
registry_password:
description: “Password for image registry”
required: true
runs:
using: "composite"
steps:
- uses: docker/setup-buildx-action#v1
- uses: docker/login-action#v1
with:
username: ${{inputs.registry_username}}
password: ${{inputs.registry_password}}
- uses: docker/build-push-action#v2
with:
context: .
push: true
tags: user/app:latest
Old Answer
What you are looking for is composite action which help you reuse once defined set of steps.
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: octocat/say-hello#v1
- run: luarocks install luacheck
- run: ./run-linter.sh
test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: octocat/say-hello#v1
- run: luarocks install busted
- run: ./build-project.sh
- run: ./run-test-suite.sh
octocat/say-hello/action.yml:
runs:
using: "composite"
steps:
- run: echo "Nice to meet you!"
shell: pwsh
For more details you can also check's ADR here.
And why you can't simply run it once for all jobs, because each job may run on a different machine.
A job is a set of steps that execute on the same runner. By default, a workflow with multiple jobs will run those jobs in parallel. You can also configure a workflow to run jobs sequentially. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.
There are 3 main approaches for code reusing in GitHub Actions:
Reusable Workflows
Dispatched workflows
Composite Actions
There is an article describing their pros and cons.
In your case if the duplicated steps are in the single workflow you also can:
extract them to the "preparation" job
upload build artifacts
add "preparation" job to "needs" key of both jobs
download build artifact in both jobs
The below code snippet should do the work for you. Instead of creating two jobs, one for lint and another for test, you can use one job and create multiple tasks
name: Github Actions CI
on: [ pull_request ]
jobs:
lint:
name: Lint and Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: leafo/gh-actions-lua#v8.0.0
- uses: leafo/gh-actions-luarocks#v4.0.0
- name: Install Luacheck
run: luarocks install luacheck
- name: Run Lint Check
- run: ./run-linter.sh
- name: Install Busted
run: luarocks install busted
- name: Build Project
run: ./build-project.sh
- name: Run Test Suite
- run: ./run-test-suite.sh
what is the difference between
steps:
- name: npm install, build, and test
run: |
npm ci
npm run build --if-present
npm test
and
steps:
- name: npm install, build, and test
- run: npm ci
- run: npm run build --if-present
- run: npm test
in github actions?
I tried to read the documentation on steps but it does not mention anything like that
Difference is that first example is executed as single script with three commands, and second example is executed as three different one-line scripts (side note: second example is invalid, as you use step with name without run, i'll ignore that line).
Let's assume for a second that npm does not create any output when running. In first example, if one of commands fails, it might be a problem to identify which one - you have only one step marked as failed. In second example, you'll know exactly where the problem is, as each command is its own step.
Let's assume for a second that npm needs to be run in specific subdirectory. We need to remember that each steps always starts in workspace directory / repo's root directory, so we need enter directory where our stuff is first.
- run: |
cd my/directory
npm ci
npm run build --if-present
npm test
- run: npm ci
working-directory: my/directory
- run: npm run build --if-present
working-directory: my/directory
- run: npm test
working-directory: my/directory
OR
- run: cd my/directory && npm ci
- run: cd my/directory && npm run build --if-present
- run: cd my/directory && npm test
Let's assume for a second npm test needs to be run only on push event, but workflow is configured to run on: [push, pull_request]
- run: |
npm ci
npm run build --if-present
if [ "${{ github.event_name }}" == "push" ]; then
npm test
fi
shell: bash
- run: npm ci
- run: npm run build --if-present
- run: npm test
if: github.event_name == 'push'
Under Actions tab, when processing pull_request event, second example will be displayed as...
- Run npm ci
- Run npm run build...
- Run npm test <-- this one will be grayed out
...and you need only a quick look to see that npm test step is skipped. In first example you'll have to expand step first and inspect log to notice any difference.
And so on, and so on, there's dozens of scenarios when it's easier/better to use all-in-one step, and as much scenarios when command-by-command steps are the way to go; it's up to you to decide which one fits you best.
At the end of the day, both examples do exactly same thing, after all. But if anything goes wrong along the way, picking one way to run commands over another (which also changes how they're displayed) can make a difference how long it gonna take to prepare a fix.
run: | is executed as a single-line script with the commands you mentioned.
Example:
run: Each command will be executed as a one-line script.