How to replace string in expression with GitHub actions - github-actions

This is my action that returns $TOXENV that looks like this py3.6-django2.2 I'd like to $TOXENV to look like this instead py36-django22 is there any substitute/replace function that I could use to replace . char?
name: CI
on:
workflow_dispatch:
branches: [ master, actions ]
jobs:
demo:
runs-on: ubuntu-latest
strategy:
matrix:
python: [3.6, 3.7, 3.8, 3.9]
django: ['2.2', '3.0']
steps:
- uses: actions/checkout#v2
- uses: actions/setup-python#v1
name: Set up Python ${{ matrix.python }} ${{ matrix.django }}
with:
python-version: ${{ matrix.python }}
- name: python version
env:
TOXENV: "py${{ matrix.python }}-django${{ matrix.django }}"
run:
echo $TOXENV

Another way by using BASH native variable substitution:
- name: python version
env:
TOXENV: "py${{ matrix.python }}-django${{ matrix.django }}"
run: |
TOXENV=${{ env.TOXENV }}
TOXENV=${TOXENV//.} # replace all dots
echo TOXENV=${TOXENV} >> $GITHUB_ENV # update GitHub ENV vars
- name: print env
run: echo ${{ env.TOXENV }}
The idea is to read the GitHub actions expression variable into a BASH variable and do the string manipulation then export or set-output to update in GitHub actions runtime.

I don't think there's an easy way to do this in the env directive of your step when defining the value of TOXENV. The env directive accepts expressions, but the functions that can be used in expressions are limited, with nothing that can replace arbitrary characters. The closest I could find is format(), but that unfortunately requires numbered braces in the target string, which won't work for your situation.
Instead, perhaps you could set the value of TOXENV in the run directive using sed, then add it to the environment:
- name: python version
run:
RAW_TOXENV="py${{ matrix.python }}-django${{ matrix.django }}"
TOXENV=$(echo $RAW_TOXENV | sed 's/\.//')
echo "TOXENV=$TOXENV" >> $GITHUB_ENV

Related

GitHub Actions Environment variables for matrix-based testing

I am struggling with the recent change on how GitHub Action Workflows defines runtime variables, replacing the "set-ouput" approach with environment variables.
Last summer, it took me a couple of hours to figure out the code below working for my requirement. Define a matrix of OS and python versions, so that the CI workflow can create corresponding environments and run the pytests on it.
Is there any chance to get support on how to best transform to the new approach?
env:
# JSON variables (used in our strategy/matrix)
SUPPORTED_PYTHON_VERSIONS: '\"python-version\":[\"3.8\", \"3.9\"]'
SUPPORTED_OPERATING_SYSTEMS: '\"os\":[\"ubuntu-latest\", \"macos-latest\", \"windows-latest\"]'
jobs:
# The set-env job translates the json variables to a usable format for the workflow specifications.
set-env:
runs-on: ubuntu-latest
outputs:
py-os-matrix: ${{ steps.set-matrix-vars.outputs.py-os-matrix }}
# ^ this represents:
# matrix:
# - os: [ubuntu-latest, ...]
# - python-version: [3.7, ...]
os-matrix: ${{ steps.set-matrix-vars.outputs.os-matrix }}
# ^ this represents:
# matrix:
# - os: [ubuntu-latest, ...]
steps:
- id: set-matrix-vars
run: |
echo "::set-output name=py-os-matrix::{${{ env.SUPPORTED_PYTHON_VERSIONS }},${{ env.SUPPORTED_OPERATING_SYSTEMS }}}"
echo "::set-output name=os-matrix::{${{ env.SUPPORTED_OPERATING_SYSTEMS }}}"
test-run:
name: test on ${{ matrix.os }} - ${{ matrix.python-version }}
needs: set-env
strategy:
fail-fast: true
matrix: ${{ fromJson(needs.set-env.outputs.py-os-matrix) }}
Meanwhile, a more elegant way of using the matrix keyword functions is available. My new implementation is much cleaner and, most important, avoids the above-mentioned deprecated function of printing env variables to stdout. However, the point is open how to integrate python versions >=3.10 ...
jobs:
test-run:
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
python-version: [3.8, 3.9]
defaults:
run:
shell: bash
runs-on: ${{ matrix.os }}
steps:
- name: Check out repository code
uses: actions/checkout#v3
- name: Install python
id: setup-python
uses: actions/setup-python#v4
with:
python-version: ${{ matrix.python-version }}
check-latest: true
Every further step which comes after the one with setup-python is executed in the currenlty selected combination of OS and python.

GitHub Actions - How to use environment variable at job level?

I want to use environment variables at the job level. Is there a way to do it?
env:
stageEnv: UAT
jobs:
name: Upload Build
if: ${{ env.stageEnv == 'UAT' }}
steps:
....
I get unrecognized named-value: 'env' error. Tried $stageEnv and ${{ env.stageEnv }}
Note: It works when I access within 'steps', but would like this to be accessible at 'jobs' level.
I'm afraid not, but you can do like this:
env:
stageEnv: UAT
jobs:
build:
name: Build
runs-on: ubuntu-latest
outputs:
stageEnv: ${{ steps.init.outputs.stageEnv }}
steps:
- name: Make environment variables global
id: init
run: |
echo "stageEnv=${{ env.stageEnv }}" >> $GITHUB_OUTPUT
And use it in another job:
upload:
name: Upload build
needs: build
if: ${{ needs.build.outputs.stageEnv == 'UAT' }}
Note this is just an example, and I personally prefer environment variables uppercase and output variables lowercase

Setting env varables for entire workflow [duplicate]

I'd like to define and set environment variable between jobs inside my Github Actions Workflow.
The workflow below is what I've tried but unfortunately the environment variable GIT_PR_SHA_SHORT and E2E_GIT_PR_SHA not working.
Is it possible?
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
GIT_PR_SHA: ${{github.event.pull_request.head.sha}}
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
ENV_NAME: test
E2E_GIT_PR_SHA: "${{ env.ENV_NAME }}-${{ env.GIT_PR_SHA_SHORT }}"
jobs:
first-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: First Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
second-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Second Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
You reference a workflow's environment variables with ${{ env.VARIABLE_NAME }} not ${VARIABLE_NAME}. The latter is bash syntax, but these are not shell environment variables, they're workflow environment variables. They're part of the workflow execution, not part of the shell's context.
To reference a workflow environment variable:
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
one: 1
two: zwei
three: tres
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
(I like to use lower-case for my workflow environment variables, and UPPER_CASE for my shell environment variables, so that it's more obvious to me which is which.)
Similarly, this won't work:
env:
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
This is mixing bash syntax :0:10 with the workflow syntax, but the workflow variables are not run through any shell. No virtual machine has been started when the workflow file is parsed, so there's no shell to run things though.
If you wanted to use bash expressions to manipulate the environment, you would need to create a step that runs bash to do that, and you would need to use the ::set-env or ::set-output syntax.
Then you can refer to a step's output using the ${{ steps... }} context.
Unfortunately, passing things between different jobs is trickier, since they run on different virtual machines. You'll need to set variables on the overall workflow itself. You'll need to first ::set-output so that it's visible to the job, then you can raise the visibility from the job to the workflow.
name: Demonstration
on:
push:
branches: [master]
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- id: identify
run: |
# use bash variable expression to get the substring
export GIT_PR_SHA="${{ github.sha }}"
export GIT_PR_SHA_SHORT="${GIT_PR_SHA:0:10}"
echo "::set-output name=git_pr_sha::${GIT_PR_SHA}"
echo "::set-output name=git_pr_sha_short::${GIT_PR_SHA_SHORT}"
outputs:
git_pr_sha: ${{ steps.identify.outputs.git_pr_sha }}
git_pr_sha_short: ${{ steps.identify.outputs.git_pr_sha_short }}
second-job:
needs: first-job
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ needs.first-job.outputs.git_pr_sha }}"
echo "${{ needs.first-job.outputs.git_pr_sha_short }}"
I'd like to add an extension to this since I've had similar difficulties finding how to compute & set environment variables for multi-step use.
Below is a basic example of how to push back to the github environment from within a step if processing is needed to compute an environment variable for later use. You can also update existing variables this same way, not just create new.
name: minimal variable example
on:
push:
env:
MAJOR: "1"
MINOR: "0"
PATCH: "1"
jobs:
vars-example:
runs-on: ubuntu-latest
steps:
- name: only available local variable
run: LOCAL_VERSION=${MAJOR}.${MINOR}.${PATCH}
- name: available across multiple steps
run: echo "GLOBAL_VERSION=${MAJOR}.${MINOR}.${PATCH}" >> $GITHUB_ENV
- name: Vars
run: |
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
which results in Vars output of
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
shell: /usr/bin/bash -e {0}
env:
MAJOR: 1
MINOR: 0
PATCH: 1
GLOBAL_VERSION: 1.0.1
LOCAL_VERSION =
GLOBAL_VERSION = 1.0.1
You can't use env in an expression under the env element. I don't see another way than using duplicate values instead of an expression.
The env context syntax allows you to use the value of an environment variable in your workflow file. You can use the env context in the value of any key in a step except for the id and uses keys. For more information on the step syntax, see "Workflow syntax for GitHub Actions".
https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#env-context
The following is the newer way of sharing the variables across jobs after the deprication
Have also covered the scenario when the value itself is a variable and needs to be computed at runtime.
The following would be invoked when a push is made master branch
name: Sharing envs across jobs
on:
push:
branches: ['master']
env:
one: onevalue
two: twovalue
three: threevalue
jobs:
job0:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
job1:
runs-on: ubuntu-latest
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: echo "test=$(date +"%d-%m-%Y")-asdfads223" >> $GITHUB_OUTPUT
- id: step2
run: echo "test=world" >> $GITHUB_OUTPUT
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}

GitHub Action - Define Workflow Level Environment Variable Between Jobs

I'd like to define and set environment variable between jobs inside my Github Actions Workflow.
The workflow below is what I've tried but unfortunately the environment variable GIT_PR_SHA_SHORT and E2E_GIT_PR_SHA not working.
Is it possible?
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
GIT_PR_SHA: ${{github.event.pull_request.head.sha}}
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
ENV_NAME: test
E2E_GIT_PR_SHA: "${{ env.ENV_NAME }}-${{ env.GIT_PR_SHA_SHORT }}"
jobs:
first-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: First Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
second-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Second Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
You reference a workflow's environment variables with ${{ env.VARIABLE_NAME }} not ${VARIABLE_NAME}. The latter is bash syntax, but these are not shell environment variables, they're workflow environment variables. They're part of the workflow execution, not part of the shell's context.
To reference a workflow environment variable:
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
one: 1
two: zwei
three: tres
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
(I like to use lower-case for my workflow environment variables, and UPPER_CASE for my shell environment variables, so that it's more obvious to me which is which.)
Similarly, this won't work:
env:
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
This is mixing bash syntax :0:10 with the workflow syntax, but the workflow variables are not run through any shell. No virtual machine has been started when the workflow file is parsed, so there's no shell to run things though.
If you wanted to use bash expressions to manipulate the environment, you would need to create a step that runs bash to do that, and you would need to use the ::set-env or ::set-output syntax.
Then you can refer to a step's output using the ${{ steps... }} context.
Unfortunately, passing things between different jobs is trickier, since they run on different virtual machines. You'll need to set variables on the overall workflow itself. You'll need to first ::set-output so that it's visible to the job, then you can raise the visibility from the job to the workflow.
name: Demonstration
on:
push:
branches: [master]
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- id: identify
run: |
# use bash variable expression to get the substring
export GIT_PR_SHA="${{ github.sha }}"
export GIT_PR_SHA_SHORT="${GIT_PR_SHA:0:10}"
echo "::set-output name=git_pr_sha::${GIT_PR_SHA}"
echo "::set-output name=git_pr_sha_short::${GIT_PR_SHA_SHORT}"
outputs:
git_pr_sha: ${{ steps.identify.outputs.git_pr_sha }}
git_pr_sha_short: ${{ steps.identify.outputs.git_pr_sha_short }}
second-job:
needs: first-job
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ needs.first-job.outputs.git_pr_sha }}"
echo "${{ needs.first-job.outputs.git_pr_sha_short }}"
I'd like to add an extension to this since I've had similar difficulties finding how to compute & set environment variables for multi-step use.
Below is a basic example of how to push back to the github environment from within a step if processing is needed to compute an environment variable for later use. You can also update existing variables this same way, not just create new.
name: minimal variable example
on:
push:
env:
MAJOR: "1"
MINOR: "0"
PATCH: "1"
jobs:
vars-example:
runs-on: ubuntu-latest
steps:
- name: only available local variable
run: LOCAL_VERSION=${MAJOR}.${MINOR}.${PATCH}
- name: available across multiple steps
run: echo "GLOBAL_VERSION=${MAJOR}.${MINOR}.${PATCH}" >> $GITHUB_ENV
- name: Vars
run: |
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
which results in Vars output of
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
shell: /usr/bin/bash -e {0}
env:
MAJOR: 1
MINOR: 0
PATCH: 1
GLOBAL_VERSION: 1.0.1
LOCAL_VERSION =
GLOBAL_VERSION = 1.0.1
You can't use env in an expression under the env element. I don't see another way than using duplicate values instead of an expression.
The env context syntax allows you to use the value of an environment variable in your workflow file. You can use the env context in the value of any key in a step except for the id and uses keys. For more information on the step syntax, see "Workflow syntax for GitHub Actions".
https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#env-context
The following is the newer way of sharing the variables across jobs after the deprication
Have also covered the scenario when the value itself is a variable and needs to be computed at runtime.
The following would be invoked when a push is made master branch
name: Sharing envs across jobs
on:
push:
branches: ['master']
env:
one: onevalue
two: twovalue
three: threevalue
jobs:
job0:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
job1:
runs-on: ubuntu-latest
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: echo "test=$(date +"%d-%m-%Y")-asdfads223" >> $GITHUB_OUTPUT
- id: step2
run: echo "test=world" >> $GITHUB_OUTPUT
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}

GitHub Actions expression functions: string manipulation?

In a GitHub Actions workflow definition file, there's a set of built-in functions that you can use in expressions.
For example: ${{ toJson(github) }}
Are there any string manipulation functions that can be used in an expression, such as toLowerCase?
The documentation page doesn't mention any. However, I'm wondering if Github uses some sort of standard templating / expression eval library under the hood which has provides a larger set of functions out of the box.
Impossible. GitHub expressions doesn't allow string modification, only concatenation.
You could do almost the same with a custom step in a build job, but this means that you won't be able to use that variable everywhere (for example "processed" environment name is out of the question).
env:
UPPERCASE_VAR: "HELLO"
steps:
- id: toLowerCase
run: INPUT=${{ env.UPPERCASE_VAR }} echo "::set-output name=lowerCaseValue::${INPUT,,}"
- run: echo ${{steps.toLowerCase.outputs.lowerCaseValue}}
I wanted to replace some chars in git version strings and was able to make a step like so:
- name: prepare version string
id: prep_version
run: |
export test_version=$(echo ${{ steps.tag_version.outputs.new_version }} | sed 's/[^0-9,a-z,A-Z]/\-/g')
echo ::set-output name=version::$test_version
that worked pretty good for me... so really we have anything that we could put on a cmd line
I usually start by setting all global variables that I will use throughout the workflow.
jobs:
variables:
outputs:
tag_name: ${{ steps.var.outputs.tag_name}}
runs-on: "ubuntu-latest"
steps:
- name: Setting global variables
uses: actions/github-script#v6
id: var
with:
script: |
core.setOutput('tag_name', '${{ github.head_ref }}'.toLowerCase().replaceAll(/[/.]/g, '-').trim('-'));
Usage:
deploy:
needs: [build, variables]
runs-on: [ self-hosted, preview ]
env:
TAG_NAME: ${{ needs.variables.outputs.tag_name }}
step:
-name: Echo variables
run: |
echo ${{ needs.variables.outputs.tag_name }}
echo ${{ env.TAG_NAME}}