Can an OpenShift CronJob benefit from ImageStream? - openshift

I have my CronJob working fine without the use of an image stream.
The job runs every 15 minutes and always pulls a tagged image, e.g. my-cron:stable.
Since the image is always pulled and the schedule tells the cluster when to run my job, what do I gain from knowing that there's an updated version of my image?
If the image changes and there's a running instance of my job, I want the job to complete using the current version of the image.
In the next scheduled run the updated image is pulled (AlwaysPull). So it seems I don't gain much tracking changes to an image stream for cron jobs.

ImageStream triggers only BuildConfigs and DeploymentConfigs, as per https://docs.openshift.com/container-platform/4.7/openshift_images/image-streams-manage.html .
Upstream kubernetes doesn't have a concept of ImageStream, so there is no triggering for 'vanilla' resource types. CronJob is used both in openshift and kubernetes (apiVersion: batch/v1beta1), and AFAIK the only way to access an imagestream is to use full path to internal registry, which is not that convenient. Your cronjob won't restart or won't be stopped for some reason, if imagestream is updated, because from kubernetes standpoint the image is pulled only when cronjob has been triggered, and after that it just waits for a job to complete.
As i see it - you are not gaining much from using imagestreams, because one of the main points, ability to use triggers, is not usable for cronjobs. The only reason to use it in CronJobs is if you are pushing directly to internal registry for some reason, but that's a bad practice too.
See following links for reference:
https://access.redhat.com/solutions/4815671
How to specify OpenShift image when creating a Job
Quoting redhat solution here:
Resolution
When using an image stream inside the project to run a cronjob,
specify the full path of the image:
[...]
spec:
jobTemplate:
spec:
template:
spec:
containers:
image: docker-registry.default.svc:5000/my-app-namespace/cronjob-image:latest
name: cronjob-image
[...]
Note that you can also put the ':latest' (or a specific tag) after the
image.
In this example, the cronjob will use the imagestream cronjob-image
from project my-app-namespace:
$ oc get is -n my-app-namespace [...]
imagestream.image.openshift.io/cronjob-image
docker-registry.default.svc:5000/my-app-namespace/cronjob-image
latest 27 minutes ago
Root Cause
The image was specified without its full path to the internal docker
registry. If the full path is not used (i.e. putting only
cronjob-image, OpenShift won't be able to find it.[...]

By using an ImageStream reference, you can avoid having to include the container image repository hostname hostname and port, and the project name in your Cron Job definition.
The docker repository reference looks likes this:
image-registry.openshift-image-registry.svc:5000/my-project/my-is:latest
The value of the equivalent annotation placed on a Cron Job looks like this:
[
{
"from": {
"kind": "ImageStreamTag",
"name": "my-is:latest"
},
"fieldPath": "spec.jobTemplate.spec.template.spec.containers[?(#.name==\"my-container\")].image"
}
]
On the one hand, this is longer. On the other hand, it includes less redundant information.
So, compared to other types of kubernetes resources, Image Streams don't add a great deal of functionality to Cron Jobs. But you might benefit from not having to hardcode the project name if for instance you kept the Cron Job YAML in Git and wanted to apply it to several different projects.
Kubernetes-native resources which contain a pod can be updated automatically in response to an image stream tag update by adding the image.openshift.io/triggers annotation.
This annotation can be placed on CronJobs, Deployments, StatefulSets, DaemonSets, Jobs, ReplicationControllers, etc.
The easiest way to do this is with the oc command.
$ oc set triggers cronjob/my-cronjob
NAME TYPE VALUE AUTO
cronjobs/my-cronjob config true
$ oc set triggers cronjob/my-cronjob --from-image=my-is:latest -c my-container
cronjob.batch/my-cronjob updated
$ oc set triggers cronjob/my-cronjob
NAME TYPE VALUE AUTO
cronjobs/my-cronjob config true
cronjobs/my-cronjob image my-is:latest (my-container) true
The effect of the oc set triggers command was to add the annotation to the CronJob, which we can examine with:
$ oc get cronjob/my-cronjob -o json | jq '.metadata.annotations["image.openshift.io/triggers"]' -r | jq
[
{
"from": {
"kind": "ImageStreamTag",
"name": "my-is:latest"
},
"fieldPath": "spec.jobTemplate.spec.template.spec.containers[?(#.name==\"my-container\")].image"
}
]
This is documented in Images - Triggering updates on image stream changes - but the syntax in the documentation appears to be wrong, so use oc set triggers if you find that the annotation you write by hand doesn't work.

Related

Configuring CodeQL with Github actions using well known weaknesses

I am new to CodeQL and therefore my apologies if my question is an obvious one, however, I've been unable to understand a few simple concepts.
Firstly, I can easily configure a public repo with a github action using a yml file configured as follows:
on:
push:
branches: [ master ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ master ]
jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [ 'java' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
steps:
- name: Checkout repository
uses: actions/checkout#v3
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init#v2
with:
queries: +security-extended
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild#v2
# ℹī¸ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
# - run: |
# echo "Run, Build Application using script"
# ./location_of_script_within_repo/buildscript.sh
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze#v2
As indicated in the yaml file, I'm using Java as the language. What I'm trying to then do is trigger a failure / alert with a simple code such as this in Java.
public class Main {
public static void main(String[] args) {
// Example code for https://cwe.mitre.org/data/definitions/476.html
String cmd = System.getProperty("cmd");
cmd = cmd.trim();
}
}
This simple code is an example from Common Weakness Enumeration (CWE) 416 where I'm trying to dereference a variable that hasn't been defined.
If I go to Security -> Code scanning alerts it will show that the scanning was performed but not alerts were found.
Basically, I'm wondering if I need to initialize the CodeQL with a specific CWE under the Initialize CodeQL step in the yaml file.
CodeQL only has a specific set of queries, which do not cover all possible CWEs. This list shows the currently covered CWEs for Java.
As far as I know there exists no query at the moment which detects the specific issue you are showing in your question (there are however queries which detect derefencing null). The reason for this is most likely that it would be difficult to prevent false positives. For example if your application is started with -Dcmd, then the system property would not be null. Similarly there could be a call to System.setProperty in a different part of the application which sets the system property to a non-null value.
Besides that you have configured queries: +security-extended but the type of the query you are looking for (assuming it existed) would most likely be in the query suite security-and-quality because it is not directly security related.
You could also try to write your own queries and then include them in the code scanning workflow. Some concepts of CodeQL might feel a bit unfamiliar at first, but they provide great examples and tutorials for getting started. However, you should probably first check if the provided queries already suffice for your use case.
Since May 2022:
Using CodeQL query packs (and its associated CWE coverage, with query specifiers) is still beta, but not going anywhere
its setup has been simplified:
Code scanning can be set up more easily without committing a workflow file to the repository (Jan. 2023)
Code scanning's new default setup feature automatically finds and sets up the best CodeQL configuration for your repository.
This will detect the languages in the repository and enable CodeQL analysis for every pull request and every push to the default branch and any protected branches.
Default setup currently supports analysis of JavaScript (including TypeScript), Python, and Ruby code.
More languages will be supported soon, and all other languages supported by CodeQL continue to work using a GitHub Actions workflow file.
The new default setup feature is available for CodeQL on repositories that use GitHub Actions.
You can use default setup on your repository's "Settings" tab under "Code security and analysis" (accessible by repository admins and security managers).
The options to set up code scanning using an Actions workflow file or through API upload from 3rd party CI/CD systems remain supported and are unchanged.
This more advanced setup method can be useful if you need to alter the default configuration, for example to include custom query packs.
Default setup configurations can also be converted to advanced setups if your analysis requirements change.
Default setup is currently available at the repository level.
We are actively working on future features at the organization level so you can easily set up code scanning at scale across large numbers of repositories.
This has shipped to GitHub.com and will be available in GitHub Enterprise Server 3.9.
To learn more, read the documentation on setting up code scanning for a repository.
In your case, you would still need an Actions workflow file, to specify a query pack.

In GitHub Actions is it possible to pass steps to a sub-action?

Is there any way that a step in a workflow job can specify sub-steps to pass to a custom action?
steps:
- uses: ...
with:
steps:
- ...
- ...
- ...
For example, a uses: actions/cache#v2 will attempt to download a snapshot immediately (in the current step), and also inserts a post step to upload a fresh snapshot after all other steps in the parent job. This works well for build processes that (through multiple stages) intelligently recognise which objects need to be recreated and which do not. But it is not suited for workflows that need to set up a clean environment (or generate test data) that may then be manipulated by the tests. I'd like to make a different cache action, that is explicitly passed instructions for how to regenerate the cache from scratch, and which uploads the state before returning to the following step of the parent action.
Is there any way this could be achieved? (Composite actions? Advanced yaml syntax? Template expressions? Low-level actions toolkits/features? Implementing an interpreter to re-parse the input parameter?)

Rancher - Is it possible to spin-up / recreate an entire namespace available in one environment on a new environment

Rancher: v2.2.4
In Rancher GUI, I see on one of our environment (Dev) and it contains a namespace 'n1'. This namespace under different sections (i.e. Workloads, LoadBalancers, ConfigMaps, Volumes etc) have few entries (containers/settings etc).
I want to create the same namespace on a new environment where Rancher is running. This environment lets say is (Test). After getting all the required docker images (sudo docker image pull <server:port>/<imagename:imageversion>), do I need to download YAMLs of all these entries under each sections and import them to the target environment? (possibly changing volumes-id, container image entries i.e. name: <server:port>/<imagename:imageversion> locations (if any), controller-uid to keep the one on the target (TEST) environement)? My understanding is, if I create a new workload/add anything under a respective section, the label/annotations will generate a fresh controller-id value! so, I'm wondering before importing the YAML, if I should leave the controller-uid entry value blank (not sure if it'll barf).
Is there a simple way to spin up/create an entire namespace 'n1' on TEST environment (i.e. namespace replica of n1 Dev in Test) with auto-generating the necessary Storage bits (volume classes/volumes and persistent volumes - all of these have some Vol ID/name/uid associated with each entity), Deployments bits (uid/controller-uids) etc?
What's an efficient way to do this so that I don't have to manually download YAMLs (from Dev) and import them in Test at each component level (i.e. Volumes YAMLs, Volume Class YAML, Workloads/Deployment YAMLs etc - one by one)?
You can use the following to grab all resources from a namespace and apply them in a new namespace.
#!/bin/bash
SourceNamespace="SourceNS"
TargetNamespace="TargetNS"
TempDir="./tmp"
echo "Grabbing all resources in $SourceNamespace"
for APIResource in `kubectl api-resources --verbs=list --namespaced -o name`
do
kubectl -n "$SourceNamespace" get "$APIResource" -o yaml > "$TempDir"/"$APIResource".yaml
done
echo "Deploying all resources in $TargetNamespace"
for yaml in `ls $TempDir`
do
kubectl apply -n "$TargetNamespace" -f "$TempDir"/"$yaml"
done

Artifactory REST API: how to add/append docker image artifacts to Build Info?

I have a bamboo server that builds docker images and pushes them to artifactory via reverse proxy. Since the push from the reverse proxy (NGINX) does not provide build information, I need to create it for further requirements.
I came across this REST API https://www.jfrog.com/knowledge-base/how-to-create-and-deploy-a-build-info-json-file-via-the-rest-api/
That went very fine, but the problem is that, when I go to the web UI and check for the build, for every artifact it appears only the name, and even the type, but the "Repo Path" column, shows "No path found (externally resolved or deleted/overwritten)".
Via appending artifacts to an existing module (/api/build/append), or creating a new build (like in the link above), my modules section looks like:
[{
"id":"my-module",
"artifacts":[
{
"type": "txt",
"sha1": "73f4617b4a64dc95c63581609ea0734aab987d98",
"md5": "00ebbb0b0444645b9489825d7a11c4bf",
"name": "MyArtifact.txt"
}
]
}]
The artifact json was created looking at the attributes of the artifacts pushed by the reverse proxy.
In short, I want to create build info linked to artifacts that are already uploaded into artifactory.
Any ideas why this isn't correctly linking the artifacts to the
build info?
Do I need any extra attributes?
Is it really possible to do what I'm looking for?
I'm not using JFrog's artifactory client for my task. I'm just using Advanced Rest Client for testing commands, and jersey client from my java classes.
I ran into the same problem and finally found some of their source code. It appears that artifacts are linked via properties.
So If your build info name is Test and the build info number is 1
You need to add the following properties to MyArtifact.txt
build.name = Test
build.number = 1
I also believe that you need to publish your build info after these properties have been added to the artifact

Configure or Create hudson job automatically

Is there any way to create new Hudson job by one more Hudson job based one previous Jobs?
For example if I need to create new bunch of jobs one by one, Automatically create 4 jobs with similar configuration with different parameter
Basically steps like this
create SVN branch I can call svn cp command and make it parametrized using script
Create some build based on new svnbranch name
Later tag it
Or other word, I need to clone the previous job and give the new branch name where ever $ Branch comes in new job.
Thanks
You can try the Hudson Remote API for this kind of task (setting up an Hudson project).
See this tutorial for instance, and remember you can display the help quite easily:
java -jar hudson-cli.jar -s http://your_Hudson_server/ help
So, to copy a job:
java -jar hudson-cli.jar -s http://your_Hudson_server/ copy-job myjob copy-myjob
You could use groovy system script like this :
def jenkins = hudson.model.Hudson.instance
def template = jenkins.getItem("MyTemplate")
def job = jenkins.copy(template,"MyNewJob")
job.scm = new hudson.scm.SubversionSCM("http://base/branches/mybranche")
job.save()
Kind of already covered in the other answers, but for an easy way to copy the config.xml over:
curl --user USER:PASS -H "Content-Type: text/xml" -s
--data-binary "#config.xml" "http://hudsonserver:8080/createItem?name=newjobname"
There seems to be a plugin for jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
I have not tested the plug-in yet. But if the plugin works, it should alleviate some of human errors from straight copying a job and modifying variables/values.
def jenkins = hudson.model.Hudson.instance
def template = jenkins.getItem("MyTemplate")
def job = jenkins.copy(template,"MyNewJob")
job.save()
I used this now I have to change the parameter values of MyNewJob using Groovy how will I do that?
ex I have a parameter called "Build_BranchName" and the default is //perforce/mybranch
I have to change it to
//perforce/mynewbranch
You have the option that VonC just gave you (which is probably the safest way but you can also go a different rout by just creating a new directory in {Hudson_Home}\jobs (the directory name will be the job name) and copy a modified config.xml in there. The modification will basically just be the SVN URL. You should check out the xml from the job that you are copying. You need to find out how you change the xml file via script, but this is a secondary problem.
Unfortunately, you have to either restart Hudson, or force a reload of the configuration. (visit the page http://:/reload to reload the config).
In case you're willing to use GIT (like I do, mirroring the main SVN repo, onto the Hudson/Jenkins server, and it works great)....
..you could try Stephen Haberman's post-receive-hudson:
This hook creates new jobs for each
branch in the Hudson continuous
integration tool. Besides creating the
job if needed, the user who pushed is
added to the job's email list if they
were not already there.
In any case, that script can give you new hints on how to remote control Jenkins(Hudson).