I am struggling with the recent change on how GitHub Action Workflows defines runtime variables, replacing the "set-ouput" approach with environment variables.
Last summer, it took me a couple of hours to figure out the code below working for my requirement. Define a matrix of OS and python versions, so that the CI workflow can create corresponding environments and run the pytests on it.
Is there any chance to get support on how to best transform to the new approach?
env:
# JSON variables (used in our strategy/matrix)
SUPPORTED_PYTHON_VERSIONS: '\"python-version\":[\"3.8\", \"3.9\"]'
SUPPORTED_OPERATING_SYSTEMS: '\"os\":[\"ubuntu-latest\", \"macos-latest\", \"windows-latest\"]'
jobs:
# The set-env job translates the json variables to a usable format for the workflow specifications.
set-env:
runs-on: ubuntu-latest
outputs:
py-os-matrix: ${{ steps.set-matrix-vars.outputs.py-os-matrix }}
# ^ this represents:
# matrix:
# - os: [ubuntu-latest, ...]
# - python-version: [3.7, ...]
os-matrix: ${{ steps.set-matrix-vars.outputs.os-matrix }}
# ^ this represents:
# matrix:
# - os: [ubuntu-latest, ...]
steps:
- id: set-matrix-vars
run: |
echo "::set-output name=py-os-matrix::{${{ env.SUPPORTED_PYTHON_VERSIONS }},${{ env.SUPPORTED_OPERATING_SYSTEMS }}}"
echo "::set-output name=os-matrix::{${{ env.SUPPORTED_OPERATING_SYSTEMS }}}"
test-run:
name: test on ${{ matrix.os }} - ${{ matrix.python-version }}
needs: set-env
strategy:
fail-fast: true
matrix: ${{ fromJson(needs.set-env.outputs.py-os-matrix) }}
Meanwhile, a more elegant way of using the matrix keyword functions is available. My new implementation is much cleaner and, most important, avoids the above-mentioned deprecated function of printing env variables to stdout. However, the point is open how to integrate python versions >=3.10 ...
jobs:
test-run:
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [3.8, 3.9]
defaults:
run:
shell: bash
runs-on: ${{ matrix.os }}
steps:
- name: Check out repository code
uses: actions/checkout#v3
- name: Install python
id: setup-python
uses: actions/setup-python#v4
with:
python-version: ${{ matrix.python-version }}
check-latest: true
Every further step which comes after the one with setup-python is executed in the currenlty selected combination of OS and python.
Related
I want to use environment variables at the job level. Is there a way to do it?
env:
stageEnv: UAT
jobs:
name: Upload Build
if: ${{ env.stageEnv == 'UAT' }}
steps:
....
I get unrecognized named-value: 'env' error. Tried $stageEnv and ${{ env.stageEnv }}
Note: It works when I access within 'steps', but would like this to be accessible at 'jobs' level.
I'm afraid not, but you can do like this:
env:
stageEnv: UAT
jobs:
build:
name: Build
runs-on: ubuntu-latest
outputs:
stageEnv: ${{ steps.init.outputs.stageEnv }}
steps:
- name: Make environment variables global
id: init
run: |
echo "stageEnv=${{ env.stageEnv }}" >> $GITHUB_OUTPUT
And use it in another job:
upload:
name: Upload build
needs: build
if: ${{ needs.build.outputs.stageEnv == 'UAT' }}
Note this is just an example, and I personally prefer environment variables uppercase and output variables lowercase
I created secrets in github actions and trying to use them in reusable workflow, but I am unable to make it work, However, If I pass secrets hardcoded from caller file, it works just fine
## set_env.yml
name: Sent Env Creds and Vars
on:
push:
branches:
- main
- dev
pull_request:
branches: [ main ]
jobs:
deploy-dev:
uses: ./.github/workflows/main.yml
with:
AWS_REGION: "us-east-2"
PREFIX: "dev"
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
reusable workflow = main.yml
## main.yml
name: Deploy to AWS
# Controls when the workflow will run
on:
workflow_call:
inputs:
AWS_REGION:
required: true
type: string
PREFIX:
required: true
type: string
secrets:
AWS_ACCESS_KEY_ID:
required: true
AWS_SECRET_ACCESS_KEY:
required: true
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
terraform-deploy:
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v2
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
echo Hello, Epsilon! You are in ${{ inputs.AWS_REGION }} region ${{ inputs.PREFIX }} region
for dir in $(ls -l | grep '^d' | awk '{print $9}'); do
PARENT_DIR=`pwd`
echo $dir
cd $dir
terraform init -backend-config=${PARENT_DIR}/${{ inputs.PREFIX }}-backend.tfvars
terraform validate
terraform plan -var-file=${{ inputs.PREFIX }}_vars.tfvars
## terraform apply -input=false -auto-approve -var-file=${{ inputs.PREFIX }}_vars.tfvars
cd ..
done
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
If I hardcode secrets in set_env.yml while calling main.yml like below, it just works
jobs:
deploy-dev:
uses: ./.github/workflows/main.yml
with:
AWS_REGION: "us-east-2"
PREFIX: "dev"
secrets:
AWS_ACCESS_KEY_ID: <harcoded value>
AWS_SECRET_ACCESS_KEY: <hardcoded value>
I have been trying to make it work in many ways but doesnt work. Please help
As of May 3rd 2022, this is now possible with the new keyword inherit: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onworkflow_callsecretsinherit
In the calling workflow, you tell it to inherit the secrets in the reusable workflow:
jobs:
deploy-dev:
uses: ./.github/workflows/main.yml
with:
AWS_REGION: "us-east-2"
PREFIX: "dev"
secrets: inherit
This makes the secrets available in the reusable workflow like normal:
with:
myInput: ${{ secrets.MY_SECRET }}
Note that there's no need to declare the secrets on the workflow_call trigger.
I was running into this issue. For me the culprit was the secret value in Github secrets. The secret had been created correctly, it had the correct value and name however Github actions could not find it for some reason. Deleting the secret and recreating it seems to have solved the issue though i cannot determine why
I have a package that is a core dependency of multiple other packages within my organization. My goal is to write an action to automate/ facilitate testing of these reverse dependencies. Roughly, the action should:
Trigger on a comment in a PR.
Run the unit tests of a set of reverse-dependencies with the code in that PR.
Reply to the PR with a comment about which tests failed (if any).
Steps 1 and 3 I got to work, but I’m running into issues with step 2. My current solution is to hardcode all job outputs to pass the results from step 2 to step 3, but I'm wondering if there is a way to avoid hardcoding this.
This following example workflow illustrates my problem:
name: Test
on: push
jobs:
unit-tests:
runs-on: ${{ matrix.os }}
continue-on-error: true
name: ${{ matrix.os }} (${{ matrix.pkg }})
strategy:
fail-fast: false
matrix:
# there will be more pkgs and OSes
os: [ubuntu-latest]
pkg: [pkgA, pkgB]
# how to avoid hardcoding these?
outputs:
ubuntu-latest-pkgA: ${{ steps.update-output.outputs.ubuntu-latest-pkgA }}
ubuntu-latest-pkgB: ${{ steps.update-output.outputs.ubuntu-latest-pkgB }}
steps:
- uses: actions/checkout#v2
- name: fake unit tests
run: |
exit 1 # fail all tests for now
shell: bash
- name: set error if tests fail
id: update-output
if: ${{ failure() }}
run: echo "::set-output name=${{ matrix.os }}-${{ matrix.pkg }}::error"
shell: bash
aggregate-results:
runs-on: ubuntu-latest
needs: unit-tests
steps:
- name: Aggregate results
env:
NEEDS: ${{ toJSON(needs) }}
run: echo "$NEEDS"
The job aggregate-results (inspired by this post) works nicely and prints:
{
"unit-tests": {
"result": "success",
"outputs": {
"ubuntu-latest-pkgA": "error",
"ubuntu-latest-pkgB": "error"
}
}
}
which I can use to create an informative comment. However, the job unit-tests requires me to hardcode the outputs for all combinations of os and pkg. Is there a way to do this dynamically?
I'm looking for this solution as well.
The only way to accomplish this approach is switch to re-usable workflows which allow to access one specific job context
https://docs.github.com/en/actions/learn-github-actions/contexts#jobs-context
This is my action that returns $TOXENV that looks like this py3.6-django2.2 I'd like to $TOXENV to look like this instead py36-django22 is there any substitute/replace function that I could use to replace . char?
name: CI
on:
workflow_dispatch:
branches: [ master, actions ]
jobs:
demo:
runs-on: ubuntu-latest
strategy:
matrix:
python: [3.6, 3.7, 3.8, 3.9]
django: ['2.2', '3.0']
steps:
- uses: actions/checkout#v2
- uses: actions/setup-python#v1
name: Set up Python ${{ matrix.python }} ${{ matrix.django }}
with:
python-version: ${{ matrix.python }}
- name: python version
env:
TOXENV: "py${{ matrix.python }}-django${{ matrix.django }}"
run:
echo $TOXENV
Another way by using BASH native variable substitution:
- name: python version
env:
TOXENV: "py${{ matrix.python }}-django${{ matrix.django }}"
run: |
TOXENV=${{ env.TOXENV }}
TOXENV=${TOXENV//.} # replace all dots
echo TOXENV=${TOXENV} >> $GITHUB_ENV # update GitHub ENV vars
- name: print env
run: echo ${{ env.TOXENV }}
The idea is to read the GitHub actions expression variable into a BASH variable and do the string manipulation then export or set-output to update in GitHub actions runtime.
I don't think there's an easy way to do this in the env directive of your step when defining the value of TOXENV. The env directive accepts expressions, but the functions that can be used in expressions are limited, with nothing that can replace arbitrary characters. The closest I could find is format(), but that unfortunately requires numbered braces in the target string, which won't work for your situation.
Instead, perhaps you could set the value of TOXENV in the run directive using sed, then add it to the environment:
- name: python version
run:
RAW_TOXENV="py${{ matrix.python }}-django${{ matrix.django }}"
TOXENV=$(echo $RAW_TOXENV | sed 's/\.//')
echo "TOXENV=$TOXENV" >> $GITHUB_ENV
I know that I can use Runner's Linux environment variables in GitHub Actions.
Do I have any other options to have variables and use them in workflow steps?
This is how variables are designed to work in GitHub Actions. I mean declared variables are mapped to env variables, like here:
name: Show env
on:
push:
branches:
- '*'
env:
somevar: 'lastvar'
jobs:
show:
runs-on: ubuntu-latest
steps:
- name: Is variable exported?
run: |
echo "${{ env.somevar }}"
However, you can't use them everywhere - please check this topic:
env:
pluginId: 'plugin-fn-xml-node'
on:
push:
paths:
- ${{env.pluginId}}/**
- .github/workflows/**
pull_request:
paths:
- ${{env.pluginId}}/**
- '.github/workflows/**'
jobs:
build:
env:
working-directory: ${{env.pluginId}}
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [8.x, 10.x, 12.x]
steps:
This will not work because you can't use workflow variable at job level.
So if you define variable at workflow level you should be able to use it across steps.
I added also dynamically set variable based on documentation
env:
somevar: 'lastvar'
jobs:
show:
runs-on: ubuntu-latest
steps:
- name: Is variable exported?
run: |
echo "${{ env.somevar }}"
echo "action_state=yellow" >> $GITHUB_ENV
- name: PowerShell script
# You may pin to the exact commit or the version.
# uses: Amadevus/pwsh-script#25a636480c7bc678a60bbf4e3e5ac03aca6cf2cd
uses: Amadevus/pwsh-script#v2.0.0
continue-on-error: true
with:
# PowerShell script to execute in Actions-hydrated context
script: |
Write-Host $env:somevar
Write-Host $env:action_state
- name: Read exported variable
run: |
echo "$action_state"
echo "${{ env.action_state }}"