github actions- if condition inside the for loop - github-actions

I'm tring to run the below workflow using for loop with if condition inside, but the action is parsed with an error, what is my mistake?
workflow:
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
jobs:
analysis:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Setup Cpp Check
run: sudo apt-get install cppcheck
- name: Run Static Analysis
run: cppcheck . --force --error-exitcode=1
unit_test:
# needs: Scan_changes
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Setup Ruby
run: sudo apt-get install ruby
- name: Setup Ceedling
run: sudo gem install ceedling
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files#v34
with:
dir_names: true
files: |
code/**
- name: Unit Test for Changed modules
run: |
for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
if: contains('$file', 'SomeString')
echo "$file was changed"
done
- name: Test
run: |
ceedling
and the output is
So i'm not sure what is the propore syntex for the if condition inside the fore loop

is not valid bash script code. You should do something like:
- name: Unit Test for Changed modules
run: |
for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
if [[ $file =~ "SomeString" ]]; then
echo "$file was changed"
fi
done

Related

Github workflow: Where does "npm ci" store the "node_modules"-folder

I'm wondering where npm ci saves the node_modules.
Furthermore I want to zip the node_modules. Is this even possible with git workflows or should I try a different approach? I need this workflow to deploy the codebase on git to a lamda function in AWS.
This is my code for example:
jobs:
node:
runs-on: ubuntu-latest
steps:
- name: ๐Ÿ›’ Checkout
uses: actions/checkout#v3
- name: ๐Ÿค– Setup Node
uses: actions/setup-node#v3
with:
node-version: 16.13.2
registry-url: 'https://npm.pkg.github.com'
cache: 'npm'
cache-dependency-path: '**/package.json'
- name: Cache
id: cache-dep
uses: actions/cache#v3
with:
path: |
./node_modules
key: ${{ runner.os }}-pricing-scraper-${{ hashFiles('package.json') }}
restore-keys: |
${{ runner.os }}-pricing-scraper-
- name: Config NPM
shell: bash
run: |
npm install -g npm#8.1.2
- name: โš™๏ธ Install node_modules
if: steps.cache-dep.outputs.cache-hit != 'true'
shell: bash
run: |
npm ci
build:
runs-on: ubuntu-latest
needs: [node]
steps:
- name: ๐Ÿ›’ Checkout
uses: actions/checkout#v3
- name: ๐Ÿ— Zip files and folder
if: steps.cache-dep.outputs.cache-hit != 'true'
shell: bash
run: |
echo Start running the zip script
# delete prior deployment zips
DIR="./output"
if [ ! -d "$DIR" ]; then
echo "Error: ${DIR} not found. Can not continue."
exit 1
fi
if [ ! -d "./node_modules" ]; then
echo "Error: node_modules not found. Can not continue."
exit 1
fi
rm "$DIR"/*
echo Deleted prior deployment zips
# Zip the core directories which are the same for every scraper.
zip -r "$DIR"/core.zip config node_modules shipping util
...
It's not saved in the repo
- name: Bash Test node_modules
if: steps.cache-dep.outputs.cache-hit != 'true'
shell: bash
run: |
echo "cached"
echo $(ls)
Output:
Run echo "cached"
cached
README.md build config output package-lock.json package.json scraper shipping testHandler.js util
The place where the node_modules are stored is ยด~/.npmยด
The problem was that I did 2 jobs (node, build) on 2 different VMs. So the installed node_modules were on the other VM (node-job) than where I needed them (build-job).

`GITHUB_PULL_REQUEST_BASE_REF: parameter null or not set` error in github actions

I am getting this error while trying to set a github actions. My goal is to set up a github actions that uses another template for linting and fixing SQL. Here is my github folder.
The models folder contains a single sql file (with .sql file extention). The content of the sql folder is an sql file testing.sql with the query: select a,b,c, document as doc from table.
The workflow file contains the following yml file:
on:
pull_request:
jobs:
test-check:
name: runner / sqlfluff (github-check)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: yu-iskw/action-sqlfluff#v3
id: lint-sql
with:
github_token: ${{ secrets.github_token }}
reporter: github-pr-review
sqlfluff_version: "1.2.0"
sqlfluff_command: "fix" # Or "lint"
config: "${{ github.workspace }}/.sqlfluff"
paths: '${{ github.workspace }}/models'
- name: 'Show outputs (Optional)'
shell: bash
run: |
echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'
The .sqlfluff file contains a default configuration from the following site: sqlfulff.
The workflow run is throwing the following error which I couldn't quite figure out:
I don't know what the line 15: GITHUB_PULL_REQUEST_BASE_REF: parameter null or not set means in the error. I would be glad if anyone can help with the error.
It is a parameter used by yu-iskw/action-sqlfluff action.yml in its entrypoint.sh.
SQL_FILE_PATTERN="${FILE_PATTERN:?}"
SOURCE_REFERENCE="origin/${GITHUB_PULL_REQUEST_BASE_REF:?}"
changed_files=$(git diff --name-only --no-color "$SOURCE_REFERENCE" "HEAD" -- "${SQLFLUFF_PATHS:?}" |
grep -e "${SQL_FILE_PATTERN:?}" |
xargs -I% bash -c 'if [[ -f "%" ]] ; then echo "%"; fi' || :)
Set it to the remote branch parameter github_base_ref you want to compare to (main for instance).
In your case:
on:
pull_request:
jobs:
test-check:
name: runner / sqlfluff (github-check)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: yu-iskw/action-sqlfluff#v3
id: lint-sql
with:
github_token: ${{ secrets.github_token }}
reporter: github-pr-review
sqlfluff_version: "1.2.0"
sqlfluff_command: "fix" # Or "lint"
config: "${{ github.workspace }}/.sqlfluff"
paths: '${{ github.workspace }}/models'
github_base_ref: "main" <========================
- name: 'Show outputs (Optional)'
shell: bash
run: |
echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'
(Do not include the <======... part, only the github_base_ref: "main")

Setting env varables for entire workflow [duplicate]

I'd like to define and set environment variable between jobs inside my Github Actions Workflow.
The workflow below is what I've tried but unfortunately the environment variable GIT_PR_SHA_SHORT and E2E_GIT_PR_SHA not working.
Is it possible?
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
GIT_PR_SHA: ${{github.event.pull_request.head.sha}}
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
ENV_NAME: test
E2E_GIT_PR_SHA: "${{ env.ENV_NAME }}-${{ env.GIT_PR_SHA_SHORT }}"
jobs:
first-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: First Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
second-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Second Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
You reference a workflow's environment variables with ${{ env.VARIABLE_NAME }} not ${VARIABLE_NAME}. The latter is bash syntax, but these are not shell environment variables, they're workflow environment variables. They're part of the workflow execution, not part of the shell's context.
To reference a workflow environment variable:
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
one: 1
two: zwei
three: tres
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
(I like to use lower-case for my workflow environment variables, and UPPER_CASE for my shell environment variables, so that it's more obvious to me which is which.)
Similarly, this won't work:
env:
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
This is mixing bash syntax :0:10 with the workflow syntax, but the workflow variables are not run through any shell. No virtual machine has been started when the workflow file is parsed, so there's no shell to run things though.
If you wanted to use bash expressions to manipulate the environment, you would need to create a step that runs bash to do that, and you would need to use the ::set-env or ::set-output syntax.
Then you can refer to a step's output using the ${{ steps... }} context.
Unfortunately, passing things between different jobs is trickier, since they run on different virtual machines. You'll need to set variables on the overall workflow itself. You'll need to first ::set-output so that it's visible to the job, then you can raise the visibility from the job to the workflow.
name: Demonstration
on:
push:
branches: [master]
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- id: identify
run: |
# use bash variable expression to get the substring
export GIT_PR_SHA="${{ github.sha }}"
export GIT_PR_SHA_SHORT="${GIT_PR_SHA:0:10}"
echo "::set-output name=git_pr_sha::${GIT_PR_SHA}"
echo "::set-output name=git_pr_sha_short::${GIT_PR_SHA_SHORT}"
outputs:
git_pr_sha: ${{ steps.identify.outputs.git_pr_sha }}
git_pr_sha_short: ${{ steps.identify.outputs.git_pr_sha_short }}
second-job:
needs: first-job
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ needs.first-job.outputs.git_pr_sha }}"
echo "${{ needs.first-job.outputs.git_pr_sha_short }}"
I'd like to add an extension to this since I've had similar difficulties finding how to compute & set environment variables for multi-step use.
Below is a basic example of how to push back to the github environment from within a step if processing is needed to compute an environment variable for later use. You can also update existing variables this same way, not just create new.
name: minimal variable example
on:
push:
env:
MAJOR: "1"
MINOR: "0"
PATCH: "1"
jobs:
vars-example:
runs-on: ubuntu-latest
steps:
- name: only available local variable
run: LOCAL_VERSION=${MAJOR}.${MINOR}.${PATCH}
- name: available across multiple steps
run: echo "GLOBAL_VERSION=${MAJOR}.${MINOR}.${PATCH}" >> $GITHUB_ENV
- name: Vars
run: |
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
which results in Vars output of
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
shell: /usr/bin/bash -e {0}
env:
MAJOR: 1
MINOR: 0
PATCH: 1
GLOBAL_VERSION: 1.0.1
LOCAL_VERSION =
GLOBAL_VERSION = 1.0.1
You can't use env in an expression under the env element. I don't see another way than using duplicate values instead of an expression.
The env context syntax allows you to use the value of an environment variable in your workflow file. You can use the env context in the value of any key in a step except for the id and uses keys. For more information on the step syntax, see "Workflow syntax for GitHub Actions".
https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#env-context
The following is the newer way of sharing the variables across jobs after the deprication
Have also covered the scenario when the value itself is a variable and needs to be computed at runtime.
The following would be invoked when a push is made master branch
name: Sharing envs across jobs
on:
push:
branches: ['master']
env:
one: onevalue
two: twovalue
three: threevalue
jobs:
job0:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
job1:
runs-on: ubuntu-latest
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: echo "test=$(date +"%d-%m-%Y")-asdfads223" >> $GITHUB_OUTPUT
- id: step2
run: echo "test=world" >> $GITHUB_OUTPUT
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}

setting output variables on Windows using cmd.exe as the shell

How can i set output variables when using shell: cmd on Windows? My repo is hosted at Github (not Gitlab).
The following is based on this accepted answer
jobs:
job1:
runs-on: my-windows-machine
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: |
echo '::echo::on'
echo "::set-output name=test::hello"
shell: cmd
- id: step2
run: |
echo '::echo::on'
echo "::set-output name=test::world"
shell: cmd
job2:
runs-on: my-windows-machine
needs: job1
steps:
- run: echo ok: ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}
shell: cmd
The echo stmt in job2 only shows ok: string if shell: cmd is used in the steps in job1.
As the OP Andrew concludes in the comments, output vars just is not supported with cmd.exe.
I went ahead and broke up the steps in my workflow to use shell: cmd for the things that need to be processed by cmd.exe and created a separate step (using the default shell) just to set the output vars.
As an alternative, you can see in "Github Actions, how to share a calculated value between job steps?" the $GITHUB_OUTPUT command, which can be used to define outputs for steps. The outputs can then be used in later steps and evaluated in with and env input sections.
You can see it used in "How can I use a GitHub action's output in a workflow?".
Note: the older ::set-output command has now (Oct. 2022) been deprecated.
name: 'Hello World'
runs:
using: "composite"
steps:
- id: random-number-generator
run: echo "name=random-id::$(echo $RANDOM)" >> $GITHUB_OUTPUT
shell: bash
...
jobs:
test-job:
runs-on: self-hosted
steps:
- name: Call Hello World
id: hello-world
uses: actions/hello-world-action#v1
- name: Comment
if: ${{ github.event_name == 'pull_request' }}
uses: actions/github-script#v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: 'Output - ${{ steps.hello-world.outputs.random-number }}'
})

GitHub Action - Define Workflow Level Environment Variable Between Jobs

I'd like to define and set environment variable between jobs inside my Github Actions Workflow.
The workflow below is what I've tried but unfortunately the environment variable GIT_PR_SHA_SHORT and E2E_GIT_PR_SHA not working.
Is it possible?
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
GIT_PR_SHA: ${{github.event.pull_request.head.sha}}
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
ENV_NAME: test
E2E_GIT_PR_SHA: "${{ env.ENV_NAME }}-${{ env.GIT_PR_SHA_SHORT }}"
jobs:
first-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: First Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
second-job:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Second Echo Step
run: |
echo "GIT_PR_SHA_SHORT = ${GIT_PR_SHA_SHORT}"
echo "E2E_GIT_PR_SHA = ${E2E_GIT_PR_SHA}"
You reference a workflow's environment variables with ${{ env.VARIABLE_NAME }} not ${VARIABLE_NAME}. The latter is bash syntax, but these are not shell environment variables, they're workflow environment variables. They're part of the workflow execution, not part of the shell's context.
To reference a workflow environment variable:
name: Git Pull Request Workflow
on:
workflow_dispatch:
pull_request:
branches:
- master
env:
one: 1
two: zwei
three: tres
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
(I like to use lower-case for my workflow environment variables, and UPPER_CASE for my shell environment variables, so that it's more obvious to me which is which.)
Similarly, this won't work:
env:
GIT_PR_SHA_SHORT: "${{ env.GIT_PR_SHA:0:10 }}"
This is mixing bash syntax :0:10 with the workflow syntax, but the workflow variables are not run through any shell. No virtual machine has been started when the workflow file is parsed, so there's no shell to run things though.
If you wanted to use bash expressions to manipulate the environment, you would need to create a step that runs bash to do that, and you would need to use the ::set-env or ::set-output syntax.
Then you can refer to a step's output using the ${{ steps... }} context.
Unfortunately, passing things between different jobs is trickier, since they run on different virtual machines. You'll need to set variables on the overall workflow itself. You'll need to first ::set-output so that it's visible to the job, then you can raise the visibility from the job to the workflow.
name: Demonstration
on:
push:
branches: [master]
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- id: identify
run: |
# use bash variable expression to get the substring
export GIT_PR_SHA="${{ github.sha }}"
export GIT_PR_SHA_SHORT="${GIT_PR_SHA:0:10}"
echo "::set-output name=git_pr_sha::${GIT_PR_SHA}"
echo "::set-output name=git_pr_sha_short::${GIT_PR_SHA_SHORT}"
outputs:
git_pr_sha: ${{ steps.identify.outputs.git_pr_sha }}
git_pr_sha_short: ${{ steps.identify.outputs.git_pr_sha_short }}
second-job:
needs: first-job
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ needs.first-job.outputs.git_pr_sha }}"
echo "${{ needs.first-job.outputs.git_pr_sha_short }}"
I'd like to add an extension to this since I've had similar difficulties finding how to compute & set environment variables for multi-step use.
Below is a basic example of how to push back to the github environment from within a step if processing is needed to compute an environment variable for later use. You can also update existing variables this same way, not just create new.
name: minimal variable example
on:
push:
env:
MAJOR: "1"
MINOR: "0"
PATCH: "1"
jobs:
vars-example:
runs-on: ubuntu-latest
steps:
- name: only available local variable
run: LOCAL_VERSION=${MAJOR}.${MINOR}.${PATCH}
- name: available across multiple steps
run: echo "GLOBAL_VERSION=${MAJOR}.${MINOR}.${PATCH}" >> $GITHUB_ENV
- name: Vars
run: |
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
which results in Vars output of
echo LOCAL_VERSION = $LOCAL_VERSION
echo GLOBAL_VERSION = $GLOBAL_VERSION
shell: /usr/bin/bash -e {0}
env:
MAJOR: 1
MINOR: 0
PATCH: 1
GLOBAL_VERSION: 1.0.1
LOCAL_VERSION =
GLOBAL_VERSION = 1.0.1
You can't use env in an expression under the env element. I don't see another way than using duplicate values instead of an expression.
The env context syntax allows you to use the value of an environment variable in your workflow file. You can use the env context in the value of any key in a step except for the id and uses keys. For more information on the step syntax, see "Workflow syntax for GitHub Actions".
https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#env-context
The following is the newer way of sharing the variables across jobs after the deprication
Have also covered the scenario when the value itself is a variable and needs to be computed at runtime.
The following would be invoked when a push is made master branch
name: Sharing envs across jobs
on:
push:
branches: ['master']
env:
one: onevalue
two: twovalue
three: threevalue
jobs:
job0:
runs-on: ubuntu-latest
steps:
- run: |
echo "${{ env.one }}"
echo "${{ env.two }}"
echo "${{ env.three }}"
job1:
runs-on: ubuntu-latest
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: echo "test=$(date +"%d-%m-%Y")-asdfads223" >> $GITHUB_OUTPUT
- id: step2
run: echo "test=world" >> $GITHUB_OUTPUT
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}