I made a cronjob in OpenShift 3.11 with a restartPolicy of Always. However, when I deleted this cronjob, and the associated jobs, the jobs were (and currently are, as of writing this) still running. I cannot figure out how to stop the job though, and would like to stop the job from running.
I have tried scaling down the deployment to zero pods, deleting the deployment and recreating it (redeploying), deleting the build config and then redoing the build config, deleting the entire project in Open Shift, then recreating the project, running: oc delete all -l app=app, oc delete jobs --all, and oc delete pods --all, none of which has worked so far.
Any suggestions regarding how to delete the cronjob are helpful!
EDIT:
cronjob.yaml:
kind: CronJob
apiVersion: batch/v1beta1
metadata:
name: --redacted--
namespace: --redacted--
selfLink: --redacted--
uid: 5d5cde7d-e8f6-11ea-8ec0-00505682ee91
resourceVersion: '178216471'
creationTimestamp: '2020-08-28T06:19:01Z'
spec:
schedule: 0 8 * * *
concurrencyPolicy: Allow
suspend: false
jobTemplate:
metadata:
creationTimestamp: null
spec:
template:
metadata:
creationTimestamp: null
spec:
containers:
- name: --redacted--
image: byrnedo/alpine-curl
args:
- '--insecure'
- '--location'
- >-
-H 'Authorization: Bearer --redacted--'
- http://--redacted--
- '-XPOST'
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
status:
lastScheduleTime: '2020-08-28T08:00:00Z'
oc get cronjobs returns no jobs.
Based on your updated question, i can see why oc delete all -l app=app haven't deleted your CronJob - there is no metadata.labels in it, so it wasn't selected. You can use
oc get all -lapp=app
to verify what is gona be deleted. There is one caveat though - all is actually not "everything", refer to Listing all resources in a namespace to see the supported way to get each and every resource in namespace.
But if you really removed a project - all resources should've been deleted. I can see only one possibility of CronJob existing - you may have added it to some other namespace too. You can use
oc get cronjob --all-namespaces --field-selector=metadata.name=you_cronjob_name
to search for cronjob with your name in all namespaces.
Related
So my java application is running in several pod in openshift and I want to print the podname in application logs for some business purpose. Is there any way to do so? Thanks
You should be able to expose the Pod name to the application using the Kubernetes "Downward API". This can either be done by exposing an environment variable with the Pod name, or mounting a file that contains the name.
Here's the docs for doing so with an environment variable: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api
Here's a trimmed down version of the example on that page, to highlight just the Pod name:
apiVersion: v1
kind: Pod
metadata:
name: dapi-envars-fieldref
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "sh", "-c"]
args:
- while true; do
echo -en '\n';
printenv MY_POD_NAME;
sleep 10;
done;
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
restartPolicy: Never
As you can see from the docs, there's a bunch of other context that you can expose also.
The equivalent docs for mounting a volume file can be found here: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api
I have deployed mongo stateful pods with an auto rolling strategy and below is the template for it. The deployment is successful and the pods are into Running state.
- apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
podManagementPolicy: Parallel
replicas: 3
strategy:
type: Rolling
template:
metadata:
labels:
role: mongo
environment: test
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo:4.0
imagePullPolicy: Always
command:
- mongod
- "--replSet"
- rs0
- "--bind_ip"
- 0.0.0.0
- "--smallfiles"
- "--noprealloc"
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
updateStrategy:
type: RollingUpdate
I am trying to update the image of the mongo using the following set command,
oc set image statefulset/mongo mongo=mongo:4.2 -n mongo-replica
While trying to update the image, the pods are into "CrashLoopBackOff" error. I am expecting the pods to be auto rolled back to the previous running version.
But the pods are struck in "CrashLoopBackOff" error state. I want the pods to be rolled back to the previous running version. Any suggestions here would be appreciated.
Statefulset unfortunately don't have a Rollback, but you can warranty your services using the probes, having a well configure Liveness and Readiness probes the changed version will only take the place of the running version with the probes answering an ok status.
In that way only one of your 3 replicas will crash in a failure, and you can work on it to solve the problem or manually rollback your changes, but without losing the delivery of your service.
More detail about this you can see on the k8s documentation:
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#forced-rollback
About the probes, you can get a good explanation about it here:
https://www.openshift.com/blog/liveness-and-readiness-probes
Is it possible to share a single persistent volume claim (PVC) between two apps (each using a pod)?
I read: Share persistent volume claims amongst containers in Kubernetes/OpenShift but didn't quite get the answer.
I tried to added a PHP app, and MySQL app (with persistent storage) within the same project. Deleted the original persistent volume (PV) and created a new one with read,write,many mode. I set the root password of the MySQL database, and the database works.
Then, I add storage to the PHP app using the same persistent volume claim with a different subpath. I found that I can't turn on both apps. After I turn one on, when I try to turn on the next one, it get stuck at creating container.
MySQL .yaml of the deployment step at openshift:
...
template:
metadata:
creationTimestamp: null
labels:
name: mysql
spec:
volumes:
- name: mysql-data
persistentVolumeClaim:
claimName: mysql
containers:
- name: mysql
...
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql/data
subPath: mysql/data
...
terminationMessagePath: /dev/termination-log
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
PHP .yaml from deployment step:
template:
metadata:
creationTimestamp: null
labels:
app: wiki2
deploymentconfig: wiki2
spec:
volumes:
- name: volume-959bo <<----
persistentVolumeClaim:
claimName: mysql
containers:
- name: wiki2
...
volumeMounts:
- name: volume-959bo
mountPath: /opt/app-root/src/w/images
subPath: wiki/images
terminationMessagePath: /dev/termination-log
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
The volume mount names are different. But that shouldn't make the two pods can't share the PVC. Or, the problem is that they can't both mount the same volume at the same time?? I can't get the termination log at /dev because if it can't mount the volume, the pod doesn't start, and I can't get the log.
The PVC's .yaml (oc get pvc -o yaml)
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-class: ebs
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
creationTimestamp: YYYY-MM-DDTHH:MM:SSZ
name: mysql
namespace: abcdefghi
resourceVersion: "123456789"
selfLink: /api/v1/namespaces/abcdefghi/persistentvolumeclaims/mysql
uid: ________-____-____-____-____________
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
volumeName: pvc-________-____-____-____-____________
status:
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
phase: Bound
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""
Suspicious Entries from oc get events
Warning FailedMount {controller-manager }
Failed to attach volume "pvc-________-____-____-____-____________"
on node "ip-172-__-__-___.xx-xxxx-x.compute.internal"
with:
Error attaching EBS volume "vol-000a00a00000000a0" to instance
"i-1111b1b11b1111111": VolumeInUse: vol-000a00a00000000a0 is
already attached to an instance
Warning FailedMount {kubelet ip-172-__-__-___.xx-xxxx-x.compute.internal}
Unable to mount volumes for pod "the pod for php app":
timeout expired waiting for volumes to attach/mount for pod "the pod".
list of unattached/unmounted volumes=
[volume-959bo default-token-xxxxx]
I tried to:
turn on the MySQL app first, and then try to turn on the PHP app
found php app can't start
turn off both apps
turn on the PHP app first, and then try to turn on the MySQL app.
found mysql app can't start
The strange thing is that the event log never says it can't mount volume for the MySQL app.
The remaining volumen to mount is either default-token-xxxxx, or volume-959bo (the volume name in PHP app), but never mysql-data (the volume name in MySQL app).
So the error seems to be caused by the underlying storage you are using, in this case EBS. The OpenShift docs actually specifically state that this is the case for block storage, see here.
I know this will work for both NFS and Glusterfs storage, and have done this in numerous projects using these storage type but unfortunately, in your case it's not supported
I want my deploy configuration to use an image that was the output of a build configuration.
I am currently using something like this:
- apiVersion: v1
kind: DeploymentConfig
metadata:
annotations:
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
app: myapp
name: myapp
spec:
replicas: 1
selector:
app: myapp
deploymentconfig: myapp
strategy:
resources: {}
template:
metadata:
annotations:
openshift.io/container.myapp.image.entrypoint: '["python3"]'
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
app: myapp
deploymentconfig: myapp
spec:
containers:
- name: myapp
image: 123.123.123.123/myproject/myapp-staging:latest
resources: {}
command:
- scripts/start_server.sh
ports:
- containerPort: 8000
test: false
triggers: []
status: {}
I had to hard-code the integrated docker registry's IP address; otherwise Kubernetes/OpenShift is not able to find the image to pull down. I would like to not hard-code the integrated docker registry's IP address, and instead use something like this:
- apiVersion: v1
kind: DeploymentConfig
metadata:
annotations:
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
app: myapp
name: myapp
spec:
replicas: 1
selector:
app: myapp
deploymentconfig: myapp
strategy:
resources: {}
template:
metadata:
annotations:
openshift.io/container.myapp.image.entrypoint: '["python3"]'
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
app: myapp
deploymentconfig: myapp
spec:
containers:
- name: myapp
from:
kind: "ImageStreamTag"
name: "myapp-staging:latest"
resources: {}
command:
- scripts/start_server.sh
ports:
- containerPort: 8000
test: false
triggers: []
status: {}
But this causes Kubernetes/OpenShift to complain with:
The DeploymentConfig "myapp" is invalid.
spec.template.spec.containers[0].image: required value
How can I specify the output of a build configuration as the image to use in a deploy configuration?
Thank you for your time!
Also, oddly enough, if I link the deploy configuration to the build configuration with a trigger, Kubernetes/OpenShift knows to look in the integrated docker for the image:
- apiVersion: v1
kind: DeploymentConfig
metadata:
annotations:
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
app: myapp-staging
name: myapp-staging
spec:
replicas: 1
selector:
app: myapp-staging
deploymentconfig: myapp-staging
strategy:
resources: {}
template:
metadata:
annotations:
openshift.io/container.myapp.image.entrypoint: '["python3"]'
openshift.io/generated-by: OpenShiftNewApp
creationTimestamp: null
labels:
app: myapp-staging
deploymentconfig: myapp-staging
spec:
containers:
- name: myapp-staging
image: myapp-staging:latest
resources: {}
command:
- scripts/start_server.sh
ports:
- containerPort: 8000
test: false
triggers:
- type: "ImageChange"
imageChangeParams:
automatic: true
containerNames:
- myapp-staging
from:
kind: ImageStreamTag
name: myapp-staging:latest
status: {}
But I don't want the automated triggering...
Update 1 (11/21/2016):
Configuring the trigger but having the trigger disabled (hence manually triggering the deploy), still left the deployment unable to find the image:
$ oc describe pod myapp-1-oodr5
Name: myapp-1-oodr5
Namespace: myproject
Security Policy: restricted
Node: node.url/123.123.123.123
Start Time: Mon, 21 Nov 2016 09:20:26 -1000
Labels: app=myapp
deployment=myapp-1
deploymentconfig=myapp
Status: Pending
IP: 123.123.123.123
Controllers: ReplicationController/myapp-1
Containers:
myapp:
Container ID:
Image: myapp-staging:latest
Image ID:
Port: 8000/TCP
Command:
scripts/start_server.sh
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Volume Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-goe98 (ro)
Environment Variables:
ALLOWED_HOSTS: myapp-myproject.url
Conditions:
Type Status
Ready False
Volumes:
default-token-goe98:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-goe98
QoS Tier: BestEffort
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
42s 42s 1 {scheduler } Scheduled Successfully assigned myapp-1-oodr5 to node.url
40s 40s 1 {kubelet node.url} implicitly required container POD Pulled Container image "openshift3/ose-pod:v3.1.1.7" already present on machine
40s 40s 1 {kubelet node.url} implicitly required container POD Created Created with docker id d3318e880e4a
40s 40s 1 {kubelet node.url} implicitly required container POD Started Started with docker id d3318e880e4a
40s 24s 2 {kubelet node.url} spec.containers{myapp} Pulling pulling image "myapp-staging:latest"
38s 23s 2 {kubelet node.url} spec.containers{myapp} Failed Failed to pull image "myapp-staging:latest": Error: image library/myapp-staging:latest not found
35s 15s 2 {kubelet node.url} spec.containers{myapp} Back-off Back-off pulling image "myapp-staging:latest"
Update 2 (08/23/2017):
In case, this helps others, here's a summary of the solution.
triggers:
- type: "ImageChange"
imageChangeParams:
automatic: true # this is required to link the build and deployment
containerNames:
- myapp-staging
from:
kind: ImageStreamTag
name: myapp-staging:latest
With the trigger and automatic set to true, the deployment should use the build's image in the internal registry.
The other comments relating to making the build not trigger a deploy relates to a separate requirement of wanting to manually deploy images from the internal registry. Here's more information about that portion:
The build needs to trigger the deployment at least once before automatic is set to false. So far a while, I was:
setting automatic to true
initiate a build and deploy
after deployment finishes, manually change automatic to false
manually, trigger a deployment later (though I did not verify if this deployed the older, out-of-date image or not)
I was initially trying to use this manual deployment as a way for a non-developer to go into the web console and make deployments. But this requirement has since been removed, so having build trigger deployments each time works just fine for us now. Builds can build at different branches and then tag the images differently. Deployments can then just use the appropriately tagged images.
Hope that helps!
Are you constructing the resource definitions by hand?
It would be easier to use oc new-build and then oc new-app if you really need to set this up as two steps for some reason. If you just want to setup the build and deployment in one go, just use oc new-app.
For example, to setup build and deployment in one go use:
oc new-app --name myapp <repository-url>
To do it in two steps use:
oc new-build --name myapp <repository-url>
oc new-app myapp
If you still rather use hand created resources, at least use the single step variant with the --dry-run -o yaml options to see what it would create for the image stream, plus build and deployment configuration. That way you can learn from it how to do it. The bit you currently have missing is an image stream.
BTW. It looks a bit suspicious that you have the entry point set to python3. That is highly unusual. What are you trying to do as right now it looks like you may be trying to do something in a way which may not work with how OpenShift works. OpenShift is mainly about long running processes and not for doing single docker run. You can do the latter, but not how you are currently doing it.
I have google cloud container engine setup. I wanted to spin pod of mysql with external volume.
ReplicationController:
apiVersion: v1
kind: ReplicationController
metadata:
labels:
name: mysql
name: mysql-controller
spec:
replicas: 1
template:
metadata:
labels:
name: mysql
spec:
containers:
- image: mysql
name: mysql
ports:
- name: mysql
containerPort: 3306
hostPort: 3306
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
gcePersistentDisk:
pdName: mysql-1-disk
fsType: ext4
When i run RC without external volume, MySQL works fine. It breaks with below error when i try to attach volume
Kubernetes POD Error:
Warning FailedSyncError syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 20s restarting failed container=mysql pod=mysql-controller-4hhqs_default(eb34ff46-8784-11e6-8f12-42010af00162)"
Disk (External Volume):
mysql-1-disk is the google cloud disk. I tried creating disk with both blank disk and image - ubuntu. Both failed with same error.
The error messages on mounting persistent disks are really not descriptive from my perspective. Use a blank disk based on your configuration file.
Some things to check:
Is the pdName exactly the same as in your CGE environment
Is the disk in the same availability zone (eg. europe-west1-c) as your cluster, otherwise it can't mount.
Hope this helps.
The problem that you face may be caused by using RC, not Pod to interact with the Persistent Disk.
As it's mentioned in documentation:
A feature of PD is that they can be mounted as read-only by multiple consumers simultaneously. This means that you can pre-populate a PD with your dataset and then serve it in parallel from as many pods as you need. Unfortunately, PDs can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed.
Using a PD on a pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1.
In this case, I may suggest you to run MySQL with Persistent Disks defining the disk connection in Pod configuration file. Sample configuration you may find here.