I need to do some directory grooming before my app is ready to be tested or deployed. I would like to utilize a Makefile target which calls a shell script in the repo to make this CI/CD-agnostic. One can call this target with make prepare_directory
The CI platform I am using is Github Actions. Here are the relevant parts of the workflow which is being run on new Pull Requests:
name: PR Tests
env:
GIT_TOKEN: ${{ secrets.GITHUB_TOKEN }}
jobs:
test:
name: Test
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
with:
fetch-depth: 1
- name: Prep directoy
run: make prepare_directory
Here is the relevant part of the Makefile (which works exactly as expected locally):
...
prepare_directory:
./scripts/prepare_directory.sh
clean:
#rm -Rf ./$(BUILDPREFIX)
.PHONY: all clean docker lint prep_avro $(dockerbuilds)
Here is the relevant part of the ./scripts/prepare-directory.sh script:
#!/bin/bash -e
# ...
# clone repo using https and GITHUB_TOKEN
git clone https://$GIT_TOKEN#github.com:USERNAME/REPO.git
When I try to clone using that URL, from the shell script, the script fails (along with the Github workflow pipeline) with the following error: fatal: unable to access 'https://github.com:USERNAME/REPO.git/': URL using bad/illegal format or missing URL
Does anybody know what I'm doing wrong?
You can add this action after your checkout step and GitHub can access your private repo dependancy.
Note:- Make sure to add a server's private key as a secret, public key to GitHub SSH keys and Please replace your private repo URL from https+auth_token to SSH. ssh://git#github.com/your_group/your_project.git
Below is the example
- uses: webfactory/ssh-agent#v0.4.1
with:
ssh-private-key: ${{ secrets.SSH_KEY }}
SSH_KEY is the private key secret which you created.
I've created a GitHub App and Github Action to workaround these limitations GitHub Actions Access Manager. Feel free to drop some feedback.
Workflow
This GitHub action will request an access token for a Target Repository from the App Server, authorize by the GitHub Action ID Token (JWT signed by GitHub).
The App Server requests a GitHub App Installation Token to read .github/access.yaml file in Target Repository.
The App Server reads .github/access.yaml file from Target Repository and determine which permissions should be granted to Requesting Repository, authorized by the GitHub App Installation Token from step 2..
The App Server requests a GitHub App Installation Token with granted permissions for Source Directory and send it back in response to the GitHub action from step 1..
The GitHub action sets the token as environment variable $GITHUB_ACCESS_TOKEN and as step output value ${{ steps.github-actions-access.outputs.token }}.
Further steps can then utilize this token to access resources of the Target Repository.
The URL in your prepare-directory.sh script's git clone call has a typo in it:
# clone repo using https and GITHUB_TOKEN
git clone https://$GIT_TOKEN#github.com:USERNAME/REPO.git
The URL there is halfway between an HTTPS git URL and an SSH one. The : should be a /. Assuming your $GIT_TOKEN contains just the token and nothing else, you'd also need a : before the #:
git clone https://$GIT_TOKEN:#github.com/USERNAME/REPO.git
If your $GIT_TOKEN contained the complete auth string, which for a Github App installation access token would be something like "x-access-token:ghs_..." then you wouldn't need the : following $GIT_TOKEN, you'd just have $GIT_TOKEN#github.com.
Related
Problem
I want to be able to add a submodule automatically using GitHub Actions.
I have tried using the following command (since it works fine when run locally on my computer)
name: Trigger to Add New Submodule
on:
repository_dispatch:
types: [ trigger-add-new-submodule ]
jobs:
addSubmodule:
name: Add Submodule
runs-on: [ self-hosted, windows ]
steps:
- name: Checkout
uses: actions/checkout#v3
with:
token: ${{ secrets.MY_TOKEN || secrets.GITHUB_TOKEN }}
submodules: recursive
- name: Pull
run: git pull
- name: Add Submodule
run: git submodule add URL PATH
However, when it is running through the GitHub Actions it appears to have frozen.
I am not sure if it is waiting for user input, or stuck for some other reason.
I used GitHub Hosted instead of self-hosted and got the following output.
fatal: Cannot prompt because user interactivity has been disabled.
bash: line 1: /dev/tty: No such device or address
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.com': No such file or directory
It appears to be an issue with lack of user interactivity.
I am not sure how to allow this, since I cannot use with: token: with run:
Question
Is this even the right way to go about doing this?
I couldn't find any pre-built GitHub Actions to do this, so I figured I should just use the git commands directly.
Is this the correct command to be using, or is there another command that would work better?
I have two repositories. A and B.
Inside A, I have a docker image. Let's say it's name is ghcr.io/org/a
Inside B, I have an action that wants to use this package. Both repos are private.
Here's my action code:
- name: Log in to GitHub Container Repository
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Pull the image
run: |
docker pull ghcr.io/org/a:latest
As you can see first I log into ghcr.io and I get the Login succeeded message. Then I want to pull the image from my other repo.
But I get this error:
Error response from daemon: denied
However, when I login into ghcr.io from my own machine, I have access to both repositories and I can pull any image from any private repository of mine.
Why GitHub Action from B can not pull image from A in spite of being logged in?
I have the same error even though I granted read access of package in repo a to repo b.
Starting xxx service container
/usr/bin/docker --config /home/runner/work/_temp/.docker_4a0e5f4d-beda-48eb-9580-eb88fec51e3d login ghcr.io -u user --password-stdin
/usr/bin/docker --config /home/runner/work/_temp/.docker_4a0e5f4d-beda-48eb-9580-eb88fec51e3d pull ghcr.io/onstructive/xxx/xxx:0.1.9-native
Error response from daemon: denied
Warning: Docker pull failed with exit code 1, back off 9.427 seconds before retry.
Then I figured out that my workflow in repo b, that tries to pull an image from packages in repo a, need to declare its permission.
name: Build
on:
push:
branches:
- '*'
pull_request:
branches:
- '*'
jobs:
build:
runs-on: ubuntu-latest
permissions:
packages: read # <-- this was missing and solved the problem
Did you give repo B explicit access to the package?
https://github.com/orgs/<ORG_NAME>/packages/container/<PACKAGE_NAME>/settings
I encountered this same problem, but the following worked for me:
First, as #SalTorre suggests, I needed to give org/b explicit access to the package via the interface at
https://github.com/orgs/org/packages/container/a/settings
Next, in the image repository (org/a in your case), I needed to specifically grant permission to each user (or a set of teams containing said users) who need to initiate the workflow(s) in GHA that use the image. This is because authentication uses the credentials of ${{ github.actor }}, which is the specific GitHub Actions user who initiated the workflow.
The URL for doing this is here:
https://github.com/org/a/settings/access
With these two permissions set, my workflows were able to use my custom-built containers.
I have used the [skip ci] command as mentioned here to skip workflow runs in GitHub action, where I am doing an auto-commit after an image is built in CI aciton using PAT and this works wonderfully!
But as the commit comment contains the [skip ci] command and right after that if I create a new Tag release, the CI workflow is not triggered because of the [skip ci].
Is there any way I can exclude the [skip ci] for tag push event and keep it only for one of my branches where ci action runs?
Auto commit in GitHub action:
on: push
jobs:
ci:
runs-on: ubuntu-latest
steps:
- name: build and push image
- name: update image tag
- name: Commit changes
run: |
git config --global user.name 'abc'
git config --global user.email 'xyz#users.noreply.github.com'
git remote add origin https://github.com/${{ github.repository }}
git config --global push.default current
git remote set-url origin https://${{ secrets.PERSONAL_ACCESS_TOKEN }}#github.com/${{ github.repository }}
git commit -am "build: Image tag udpated [skip ci]"
git push
As you can see this action will run for all push events and the Commit changes step will make another commit using PAT but with [skip ci] in the comment so this same workflow is not triggered
again and it works.
But when I go to release a new tag with a new Release title and description, this action doesn't get triggered.
is there a way this can be avoided?
GitHub actions uses a special kind of security token which identifies itself as GitHub actions. It uses that fact to prevent actions from triggering more actions, potentially causing a cascade.
To bypass this protection you'll need to use different security token such as a Personal Access Token or an OAuth app token to perform the tag push action.
You can store that token as an action secret.
When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run. This prevents you from accidentally creating recursive workflow runs. For example, if a workflow run pushes code using the repository's GITHUB_TOKEN, a new workflow will not run even when the repository contains a workflow configured to run when push events occur.
https://docs.github.com/en/actions/security-guides/automatic-token-authentication#using-the-github_token-in-a-workflow
What you can do is to bail out as early as possible. To do so remove the [skip ci] token from the commit message and add your own, like [do-not-build], anything will do, really.
Then add a if: condition on the job that performs the build:
if: ${{ contains(github.event.commits[0].message, '[do-not-build]') }}
It will trigger the workflow, but the immediately skip the job.
I have private repo A (which is a library) and that repo has releases. Now I have repo B which has a dependency on the artifacts of A. The dependency (i.e. which version) is stored in a json file in B. What I'm looking for is a way to download the artifacts of release X from repo A in an action/workflow in repo B.
I have seen a lengthy bash script which make this possible, but I'm wondering if there are off the shelf actions around.
If you are using a Linux runner, you can use the Fetch Github Release Asset action.
uses: dsaltares/fetch-gh-release-asset#master
with:
repo: "user/repo"
version: "tags/v1"
file: "filename.ext"
target: "targetFolder/targetFileName.ext"
token: ${{ secrets.PAT_TO_ACCESS_PRIVATE_REPO }}
Inputs
token
Required The GitHub token. Typically this will be ${{ secrets.GITHUB_TOKEN }}
file
Required The name of the file to be downloaded.
repo
The org/repo containing the release. Defaults to the current repo.
version
The release version to fetch from in the form tags/<tag_name> or <release_id>. Defaults to latest.
target
Target file path. Only supports paths to subdirectories of the GitHub Actions workspace directory
I'm currently testing Github Actions workflows on this repository.
Context
I'm trying to use this workflow (1st):
on:
workflow_dispatch:
jobs:
job:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- run: |
date > report.txt
git config user.name github-actions
git config user.email github-actions#github.com
git add .
git commit -m "generate or update report.txt file"
git push
To trigger this workflow (2nd):
on:
push:
paths:
- '**/report.txt'
pull_request:
paths:
- '**/report.txt'
jobs:
job:
runs-on: ubuntu-latest
steps:
- run: echo "Report .txt file has been updated"
What I tried
I followed the Github Action Documentation to implement the 2nd workflow with a Filter Pattern.
When I update the report.txt file locally, then commit and push the code to the repository, the 2nd workflow triggers.
However, I don't understand why the 2nd workflow doesn't trigger when the 1st workflow is completed, even with the report.txt file being updated on the default branch.
EDIT: I know I could trigger the 2nd workflow using others trigger event types (examples: repository_dispatch or workflow_run). But I'm trying to do it from a git push command on another workflow.
Question
Did I miss something on the 1st workflow to make it trigger the 2nd, or should I add something on the 2nd workflow to make it being triggered by the 1st one?
No, you didn't miss anything in your workflows.
You just need a different token.
When you use actions/checkout, it uses the GITHUB_TOKEN for authentication, and according to the documentation it doesn't trigger a new workflow run:
When you use the repository's GITHUB_TOKEN to perform tasks on behalf
of the GitHub Actions app, events triggered by the GITHUB_TOKEN will
not create a new workflow run. This prevents you from accidentally
creating recursive workflow runs.
To make it work, you need to generate a PAT (Personal Access Token), store it in your repository secrets, and use it in your checkout step:
- uses: actions/checkout#v2
with:
token: ${{ secrets.YOUR_PAT_TOKEN }}
There are three ways of authentication within a GitHub action.
1. GITHUB_TOKEN
The GITHUB_TOKEN is always available and implicitly defined by GitHub.
- uses: actions/checkout#v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
It has limited permissions, though, as it cannot trigger new workflow runs. This is why your second workflow does not start.
2. PAT (Personal Access Token)
A personal access token can be generated in your developer settings. You can then add it as an encryped secret, e.g. PAT_TOKEN to the GitHub project and use it instead of the GITHUB_TOKEN:
- uses: actions/checkout#v3
with:
token: ${{ secrets.PAT_TOKEN }}
Subsequent push actions in the same workflow will then trigger any configured GitHub workflow as if they were pushed manually.
Be aware that the personal access token has access to all of your repositories and hence can be a potential security risk.
3. Deploy key
You can generate a dedicated SSH keypair and add it to the repository as a Deploy Key. This is an alternative to the token-based authentication and has the same effect as the personal access token, but it only provides access to a single repository. It is done as follows:
Generate an SSH keypair:
ssh-keygen -N "" -f deploy_key -C "github-actions"
Add the private key (generated file deploy_key) as an encryped secret, e.g. COMMIT_KEY to the GitHub project.
Add the public key (generated file deploy_key.pub) as a deploy key with write access to the GitHub project. Tick the Allow write access checkbox.
When checking out the source code in your workflow, add the SSH key:
- uses: actions/checkout#v3
with:
ssh-key: "${{secrets.COMMIT_KEY}}"