GitActions job name based on matrix index instead of matrix value - github-actions

I'm using GitAction to run a workflow using a matrix strategy, as follows (simplified):
name: Functional Tests
...
jobs:
functional:
...
strategy:
matrix:
range:
- -e FT_FROM_IX=0 -e FT_TO_IX=300
- -e FT_FROM_IX=301 -e FT_TO_IX=600
- -e FT_FROM_IX=601 -e FT_TO_IX=900
- -e FT_FROM_IX=901 -e FT_TO_IX=1200
- -e FT_FROM_IX=1201
steps:
- uses: actions/checkout#v2
- name: Run functional test
run: |
docker run --network host -t --rm ${{ matrix.range }} -v $(pwd):/opt/fiware-orion ${{ env.TEST_IMAGE_NAME }} build -miqts functional
It works fine, but I get a ugly description at github because the matrix.range value appears as part of the job name:
I would like to have my jobs numbered (e.g. functional-1, functional-2, etc.). Is that possible using some expression to get the index of the matrix element (something like ${{ matrix.range.index }}) or any other way?
Thanks in advance!

I had a similar use case, found a simple solution:
Change matrix range to a list of objects, containing order and range.
Concatenate order with the job's name key.
Use range key as before.
Hopefully, Github Actions will add an index to the matrix jobs, simplifying the way we distinguish between them.
name: Functional Tests
...
jobs:
functional:
name: functional - ${{ matrix.payload.order }}
...
strategy:
matrix:
payload:
- { order: 1, range: '-e FT_FROM_IX=0 -e FT_TO_IX=300' }
- { order: 2, range: '-e FT_FROM_IX=301 -e FT_TO_IX=600' }
- { order: 3, range: '-e FT_FROM_IX=601 -e FT_TO_IX=900' }
...
steps:
- uses: actions/checkout#v2
- name: Run functional test
run: |
docker run --network host -t --rm ${{ matrix.payload.range }} -v $(pwd):/opt/fiware-orion ${{ env.TEST_IMAGE_NAME }} build -miqts functional

Related

setting output variables on Windows using cmd.exe as the shell

How can i set output variables when using shell: cmd on Windows? My repo is hosted at Github (not Gitlab).
The following is based on this accepted answer
jobs:
job1:
runs-on: my-windows-machine
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: |
echo '::echo::on'
echo "::set-output name=test::hello"
shell: cmd
- id: step2
run: |
echo '::echo::on'
echo "::set-output name=test::world"
shell: cmd
job2:
runs-on: my-windows-machine
needs: job1
steps:
- run: echo ok: ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}
shell: cmd
The echo stmt in job2 only shows ok: string if shell: cmd is used in the steps in job1.
As the OP Andrew concludes in the comments, output vars just is not supported with cmd.exe.
I went ahead and broke up the steps in my workflow to use shell: cmd for the things that need to be processed by cmd.exe and created a separate step (using the default shell) just to set the output vars.
As an alternative, you can see in "Github Actions, how to share a calculated value between job steps?" the $GITHUB_OUTPUT command, which can be used to define outputs for steps. The outputs can then be used in later steps and evaluated in with and env input sections.
You can see it used in "How can I use a GitHub action's output in a workflow?".
Note: the older ::set-output command has now (Oct. 2022) been deprecated.
name: 'Hello World'
runs:
using: "composite"
steps:
- id: random-number-generator
run: echo "name=random-id::$(echo $RANDOM)" >> $GITHUB_OUTPUT
shell: bash
...
jobs:
test-job:
runs-on: self-hosted
steps:
- name: Call Hello World
id: hello-world
uses: actions/hello-world-action#v1
- name: Comment
if: ${{ github.event_name == 'pull_request' }}
uses: actions/github-script#v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: 'Output - ${{ steps.hello-world.outputs.random-number }}'
})

GitHub actions: Dynamic outputs for job with strategy.matrix

I have a package that is a core dependency of multiple other packages within my organization. My goal is to write an action to automate/ facilitate testing of these reverse dependencies. Roughly, the action should:
Trigger on a comment in a PR.
Run the unit tests of a set of reverse-dependencies with the code in that PR.
Reply to the PR with a comment about which tests failed (if any).
Steps 1 and 3 I got to work, but I’m running into issues with step 2. My current solution is to hardcode all job outputs to pass the results from step 2 to step 3, but I'm wondering if there is a way to avoid hardcoding this.
This following example workflow illustrates my problem:
name: Test
on: push
jobs:
unit-tests:
runs-on: ${{ matrix.os }}
continue-on-error: true
name: ${{ matrix.os }} (${{ matrix.pkg }})
strategy:
fail-fast: false
matrix:
# there will be more pkgs and OSes
os: [ubuntu-latest]
pkg: [pkgA, pkgB]
# how to avoid hardcoding these?
outputs:
ubuntu-latest-pkgA: ${{ steps.update-output.outputs.ubuntu-latest-pkgA }}
ubuntu-latest-pkgB: ${{ steps.update-output.outputs.ubuntu-latest-pkgB }}
steps:
- uses: actions/checkout#v2
- name: fake unit tests
run: |
exit 1 # fail all tests for now
shell: bash
- name: set error if tests fail
id: update-output
if: ${{ failure() }}
run: echo "::set-output name=${{ matrix.os }}-${{ matrix.pkg }}::error"
shell: bash
aggregate-results:
runs-on: ubuntu-latest
needs: unit-tests
steps:
- name: Aggregate results
env:
NEEDS: ${{ toJSON(needs) }}
run: echo "$NEEDS"
The job aggregate-results (inspired by this post) works nicely and prints:
{
"unit-tests": {
"result": "success",
"outputs": {
"ubuntu-latest-pkgA": "error",
"ubuntu-latest-pkgB": "error"
}
}
}
which I can use to create an informative comment. However, the job unit-tests requires me to hardcode the outputs for all combinations of os and pkg. Is there a way to do this dynamically?
I'm looking for this solution as well.
The only way to accomplish this approach is switch to re-usable workflows which allow to access one specific job context
https://docs.github.com/en/actions/learn-github-actions/contexts#jobs-context

How to replace string in expression with GitHub actions

This is my action that returns $TOXENV that looks like this py3.6-django2.2 I'd like to $TOXENV to look like this instead py36-django22 is there any substitute/replace function that I could use to replace . char?
name: CI
on:
workflow_dispatch:
branches: [ master, actions ]
jobs:
demo:
runs-on: ubuntu-latest
strategy:
matrix:
python: [3.6, 3.7, 3.8, 3.9]
django: ['2.2', '3.0']
steps:
- uses: actions/checkout#v2
- uses: actions/setup-python#v1
name: Set up Python ${{ matrix.python }} ${{ matrix.django }}
with:
python-version: ${{ matrix.python }}
- name: python version
env:
TOXENV: "py${{ matrix.python }}-django${{ matrix.django }}"
run:
echo $TOXENV
Another way by using BASH native variable substitution:
- name: python version
env:
TOXENV: "py${{ matrix.python }}-django${{ matrix.django }}"
run: |
TOXENV=${{ env.TOXENV }}
TOXENV=${TOXENV//.} # replace all dots
echo TOXENV=${TOXENV} >> $GITHUB_ENV # update GitHub ENV vars
- name: print env
run: echo ${{ env.TOXENV }}
The idea is to read the GitHub actions expression variable into a BASH variable and do the string manipulation then export or set-output to update in GitHub actions runtime.
I don't think there's an easy way to do this in the env directive of your step when defining the value of TOXENV. The env directive accepts expressions, but the functions that can be used in expressions are limited, with nothing that can replace arbitrary characters. The closest I could find is format(), but that unfortunately requires numbered braces in the target string, which won't work for your situation.
Instead, perhaps you could set the value of TOXENV in the run directive using sed, then add it to the environment:
- name: python version
run:
RAW_TOXENV="py${{ matrix.python }}-django${{ matrix.django }}"
TOXENV=$(echo $RAW_TOXENV | sed 's/\.//')
echo "TOXENV=$TOXENV" >> $GITHUB_ENV

Github action optional step execution

I am trying to execute some option steps if the previous step is failing.
I need to download the old artifacts in order to avoid that my Terraform build action is not detecting changes. Therefore I added a diff action to identify if the docker file to build the zip-layer has changed, if there is no change the old artifacts from a previous execution should be downloaded. In some cases the last execution does not contain the artifacts e.g. failure of the jobs. In that case I would like to get the latest version based on the existing docker image.
Note: The code is part of a matrix execution but for simplicity I reduced the action to the problem area.
job_prepare:
.....
job_layers:
needs: job_prepare
runs-on: ubuntu-latest
strategy:
matrix: ${{fromJson(needs.job_prepare.outputs.layer_matrix)}}
steps:
- name: Checkout
uses: actions/checkout#v2
- name: matrix name
run: |
echo $GITHUB_WORKSPACE
echo ${{ matrix.path }}
- uses: technote-space/get-diff-action#v3
id: git_diff
with:
PREFIX_FILTER: ${{ matrix.prefix }}
SUFFIX_FILTER: Dockerfile
- name: LayerDockerBuild
id: layerDocker
if: steps.git_diff.outputs.diff
run: |
docker build ...
docker push ...
- name: Layer via Artifacts
if: (steps.git_diff.outputs.diff == false)
uses: dawidd6/action-download-artifact#v2
with:
workflow: review.yml
name: ${{ matrix.name }}
- name: Layer via Docker
if: steps.git_diff.outputs.diff && ${{ failure() }}
id: layerD
run: |
....
docker pull "docker.pkg.github.com/$REPO_NAME/$IMAGE_ID:$VERSION"
docker run --rm -v $GITHUB_WORKSPACE:/data docker.pkg.github.com/$REPO_NAME/$IMAGE_ID:$VERSION cp /packages/${{ matrix.name }}.zip /data
- name: Upload layer zip
if: ${{ always() }}
uses: actions/upload-artifact#v2
with:
name: ${{ matrix.name }}
path: ${{ matrix.name }}.zip
The problem is basically the logic in the line if: steps.git_diff.outputs.diff && ${{ failure() }}
thanks for your help on any hints how to make the step option when the diff is false and the step ~Layer via Artifacts~ is not failing.
To execute your step if the previous one was not a failure you can just set failure() without brackets. With the if, no need of brackets.
if: steps.git_diff.outputs.diff && failure()
Hope it will help

Map an environment variable in Github Actions

I created a GitHub Actions Job with a strategy matrix that creates a set of environment variables.
One of them is machine_architecture which is either 32 or 64.
In most steps I can use it directly i.e. via ${{ machine_architecture }}.
But some steps requires strings like 'i386' vs 'x86_64'. Is there an easy way in github actions to create a map-object that I can use in expressions like:
map_object = { 32: "i386", 64: 'x86_64' }
...
${{ map_object[machine_architecture] }}
If not, what is the idiomatic way in github actions to solve that problem?
PS: I am aware, that I can set environment variables in steps, but the problem is, that these variables are only available for the following steps (i.e. not for usage in "run-on:" tag)
In the meantime I found a solution:
Although GitHub Actions has no syntax for directly creating a Mappings/Objects it can be done indirectly with fromJson():
${{ fromJson('{ 32: "i386", 64: "x86_64" }')[machine_architecture] }}
this fromJson() will create a mapping from int to string. the following []-operator resolves the int type "machine_architecture" to a string type.
Here is a way to do it with JSON and jq. It creates the step output ${{ steps.vars.outputs.arch }} which you can use in later steps.
jobs:
varMap:
strategy:
matrix:
machine_architecture: [32, 64]
runs-on: ubuntu-latest
steps:
- name: Set arch var
id: vars
run: |
echo ::set-output name=arch::\
$(echo '{ "32": "i386", "64": "x86_64" }' | jq -r 'to_entries[] | select(.key=="${{ matrix.machine_architecture }}") | .value')
- name: Test arch var
run: echo "Testing ${{ steps.vars.outputs.arch }}"
I don't need a matrix for this, but I need a map for lookups.
Couldn't get #PeterEvans answer to work with GitHub, so I adapted it slightly:
jobs:
yourJobName:
name: Cool Job
runs-on: ubuntu-latest
steps:
- name: Get Machine Arc
run: |
MACHINE_ARC_MAP=$(cat <<END
{
32: "i386",
64: "x86_64",
999: ${{ secrets.SECRET_ARC }}
}
END
)
TARGET_ARC=$(echo $MACHINE_ARC_MAP | jq -r 'to_entries[] | select(.key=="${{ github.event.inputs.machine_architecture }}") | .value')
echo "TARGET_ARC=$TARGET_ARC" >> $GITHUB_ENV
- name: Echo Selected value
run: |
echo ${env.TARGET_VAR}
How about your "map_object" is actually a file mapping machine_architecture values into the values you need, such:
32=i386
64=x86_64
or any other format you want to keep.
Then, your job can define it in a secondary variable as:
jobs:
FirstJob:
name: job 1
runs-on: .....
steps:
- uses: ....
- name: Define variables
run: |
cat $(cat MAP_OBJECT_FILE_NAME) | grep $(cat machine_architecture)= | > MACHINE_ARCHITECTURE_STRING
From there on, you would have the MACHINE_ARCHITECTURE_STRING variable available for the jobs you need. You could of course do it much more simple concatenating or whatever, but here you maintain the mapping in your code with the mapping file and is escalable.