My customer asked me if openshift can provide the same control on memory usage as docker can, for example, docker run can have the following parameters to control memory usage when running a container:
--kernel-memory
--memory
--memory-reservation
While I searched the corresponding part in openshift, I found ResoureQuota and LimitRange should work for that, but what if a pod claims itself will use 100Mi memory by using LimitRange but actually it will consume 500Mi memory instead? the memory can still be used "illegally", seems docker with --memory can control this situation more better.
In openshift, is there any method for controlling real memory usage instead of checking what a pod claimed in LimitRange or using "oc set resources dc hello --requests=memory=256Mi"?
Best regards
Lan
As far as my experience with Openshift I have not come across the situation where the POD has consumed more memory or CPU for which it has configured. If in case it reaches the threshold, the POD automatically will be killed and restarts.
You can set the POD resource limits in the Deployment config:
resources:
limits:
cpu: 750m
memory: 1024Mi
The resources can be monitored in the metrics section of the respective POD:
Apart from the indiviual POD settings you can define your own overall project settings for each container in the POD.
$ oc get limits
NAME
limits
$ oc describe limits <NAME>
Name: <NAME>
Namespace: <NAME_SPACE>
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod memory 256Mi 32Gi - - -
Pod cpu 125m 6400m - - -
Container cpu 125m 6400m 125m 750m -
Container memory 256Mi 32Gi 512Mi 1Gi -
For more information on resource settings refer here.
If you only use --requests=memory=256Mi, you set QoS level to "burstable", which means pod can request at least 256Mi memory without upper limit except reaching project quota. If you want to limit pod memory, use --limit=memory=256Mi instead.
Related
I'm trying to run WordPress by using Kubernetes link, and the only change is I changed 20Gi to 5Gi, but when I run kubectl apply -k ., I get this error:
Error from server (Forbidden): error when creating ".": persistentvolumeclaims "wp-pv-claim" is forbidden: exceeded quota: storagequota, requested: requests.storage=5Gi, used: requests.storage=5Gi, limited: requests.storage=5Gi
I searched but did not find any related answer to mine (or even maybe I'm wrong).
Could you please answer me these questions:
How to solve the above issue?
If the volume's size is limited to 5G, then the pod cannot be bigger than 5G? I mean if I exec into the pod and run a command like dd if=/dev/zero of=file bs=1M count=8000, should it create an 8G file or not? I mean this quota and volume limits whole the pod? Or only a specific path like /var/www/html?
Edit 1
describe pvc mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: app=wordpress
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: wordpress-mysql-6c479567b-vzpm5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 4m (x222 over 59m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
I decided to summarize our comments conversation for better readability and visibility.
The issue at first seemed to be caused by resourcequota.
Error from server (Forbidden): error when creating ".": persistentvolumeclaims "wp-pv-claim" is forbidden: exceeded quota: storagequota, requested: requests.storage=5Gi, used: requests.storage=5Gi, limited: requests.storage=5Gi
It looked like there was already existing PVC and it wouldn't allow to create a new one.
OP removed the resource quota although it was not necessary in this case since the real issue was with the PVC.
kubectl describe pvc mysql-pv-claim showed the following event:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 4m (x222 over 59m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Event message:
persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Since OP created the cluster with kubeadm and kubeadm doesn't come with a predeployed storage provider out of the box; this means that it needs to be added manually. (Storage Provider is a controller that can create a volume and mount it).
Each StorageClass has a provisioner that determines what volume plugin is used for provisioning PVs. This field must be specified. Since there was no storage class in cluster, OP decided to create one and picked Local storage class but forgot that:
Local volumes do not currently support dynamic provisioning [...].
and
Local volumes can only be used as a statically created PersistentVolume. Dynamic provisioning is not supported
This means that a local volume had to be created manually.
I'd like to configure individual dataflow tasks with their own memory request and limit values. The default configuration works fine for most tasks, but we have some tasks with higher memory needs. We can start those tasks from dataflow with their own properties, overriding the default configuration. But is it possible to have a configuration in the dataflow-config on OpenShift for individual tasks? (So that we don't have to use those overriding arguments each time we start the task).
Something like this:
deployer:
kubernetes:
requests:
memory: '256Mi'
cpu: '1m'
limits:
memory: '4Gi'
cpu: '6000m'
my-individual-task:
kubernetes:
requests:
memory: '8G'
limits:
memory :'8G'
Testing it with this configuration, the "my-individual-task" had the default configuration with 256Mi-4Gi instead of 8G-8G. (I restarted the dataflow pod with the new configuration before starting the task).⠀⠀⠀⠀⠀⠀⠀
You shouldn't set the task specific properties as server level properties. Instead, you can set these properties as task launch properties when launching the task. For instance:
task launch mytask --properties "deployer.my-individual-task.kubernetes.requests.memory=8G,deployer.my-individual-task.kubernetes.limits.memory=8G"
Testing it with this configuration, the "my-individual-task" had the default configuration with 256Mi-4Gi instead of 8G-8G
In my experience, the only way to successfully change the memory and CPU for an individual task is to provide the following properties when launching the task from the Data Flow Server UI (as properties) or when using the Task Java DSL :
deployer.<application>.kubernetes.limits.cpu=1000m
deployer.<application>.kubernetes.limits.memory=1024Mi
deployer.<application>.kubernetes.requests.cpu=800m
deployer.<application>.kubernetes.requests.memory=640Mi
Note : Specifying these in the application.properties or applicaiton.yaml file does not have any effect whatsoever.
I am trying to run an sonatype/nexus3 on openshift online v3 pro. If I just use the web console to create a new app from image it assigns it only 512Mi and it dies with OOM. It did get created though and logged a lot of java output before it died of out of memory. When using the web console there doesnt appear a way to set the memory on the image. When I try to edited the yaml of the pod it doesn't let me edited the memory limit.
Reading the docs about memory limits it suggests that I can run with this:
oc run nexus333 --image=sonatype/nexus3 --limits=memory=750Mi
Then it doesn't even start. It dies with:
{kubelet ip-172-31-59-148.ec2.internal} Error: Error response from
daemon: {"message":"create
c30deb38b3c26252bf1218cc898fbf1c68d8fc14e840076710c211d58ed87a59:
mkdir
/var/lib/docker/volumes/c30deb38b3c26252bf1218cc898fbf1c68d8fc14e840076710c211d58ed87a59:
permission denied"}
More information from oc get events:
FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
16m 16m 1 nexus333-1-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned nexus333-1-deploy to ip-172-31-50-97.ec2.internal
16m 16m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-50-97.ec2.internal} pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.6.173.0.21"
16m 16m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-50-97.ec2.internal} Successfully pulled image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.6.173.0.21"
15m 15m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-50-97.ec2.internal} Created container
15m 15m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-50-97.ec2.internal} Started container
15m 15m 1 nexus333-1-rftvd Pod Normal Scheduled {default-scheduler } Successfully assigned nexus333-1-rftvd to ip-172-31-59-148.ec2.internal
15m 14m 7 nexus333-1-rftvd Pod spec.containers{nexus333} Normal Pulling {kubelet ip-172-31-59-148.ec2.internal} pulling image "sonatype/nexus3"
15m 10m 19 nexus333-1-rftvd Pod spec.containers{nexus333} Normal Pulled {kubelet ip-172-31-59-148.ec2.internal} Successfully pulled image "sonatype/nexus3"
15m 15m 1 nexus333-1-rftvd Pod spec.containers{nexus333} Warning Failed {kubelet ip-172-31-59-148.ec2.internal} Error: Error response from daemon: {"message":"create 3aa35201bdf81d09ef4b09bba1fc843b97d0339acfef0c30cecaa1fbb6207321: mkdir /var/lib/docker/volumes/3aa35201bdf81d09ef4b09bba1fc843b97d0339acfef0c30cecaa1fbb6207321: permission denied"}
I am not sure why if I use the web console I cannot assign more memory. I am not sure why running it with oc run dies with the mkdir error. Can anyone tell me how to run sonatype/nexus3 on openshift online pro?
Looking in the documentation I see that it is a Java VM solution.
When using Java 8, memory usage can be DRAMATICALLY IMPROVED using only the following 2 runtime Java VM options:
... "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap" ...
I just deployed my container (Spring Boot JAR) that consumed over 650 MB RAM. With just these two (new) options RAM consumption dropped to just 270 MB!!!
So, with these 2 runtime settings all OOM's are left far behind! Enjoy!
You may want to also follow along with the tutorial that is in the OpenShift docs https://docs.openshift.com/online/dev_guide/app_tutorials/maven_tutorial.html
I have had success deploying this in OpenShift Online Pro
Okay the mkdir /var/lib/docker/volumes/ permission denied seems to be that the image needs a /nexus-data mount and that is refused. I saw that by deploying from the web console (dies with OOM) but the edit yaml for the created pod to see the generated volume mount.
Creating the image with the following yaml using cat nexus3_pod.ephemeral.yaml | oc create -f - with the volume mount and explicit memory settings the container will now start up:
apiVersion: "v1"
kind: "Pod"
metadata:
name: "nexus3"
labels:
name: "nexus3"
spec:
containers:
-
name: "nexus3"
resources:
requests:
memory: "1200Mi"
limits:
memory: "1200Mi"
image: "sonatype/nexus3"
ports:
-
containerPort: 8081
name: "nexus3"
volumeMounts:
- mountPath: /nexus-data
name: nexus3-1
volumes:
- emptyDir: {}
name: nexus3-1
Notes
The mage sets -Xmx1200m as documented at sonatype/docker-nexus3. So if you assign memory less than 1200Mi it will crash with OOM when the heap grows over the limit. You may as well set requested and max to be the max heap side anything.
When the allocated memory was too low it crashed die just as it was setting up the DB which corrupted the db log which meant it then got in a crash loop "couldn't load 4 byte from 0 byte file" when I recreated it with more memory. It seems that with an emptyDir the files hang around between crash restarts and memory changes (that's documented behaviour I think). I had to recreate a pod with a different name to get a clean emptyDir and assigned memory of 1200Mi to get it to all start.
I'm trying to run this Zookeeper Openshift example or the equivalent kubernetes one, but I end with errors such as:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 12s 281 {default-scheduler } Warning FailedScheduling [SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "datadir-zoo-0", which is unexpected., SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "datadir-zoo-0", which is unexpected.]
or
error finding provisioning plugin for claim test/datadir-zoo-2: cannot find volume plugin for alpha provisioning
Here is my openshift template.yaml
I'm note sure but I suspect that it might be due to line volume.alpha.kubernetes.io/storage-class: anything, because I don't think that there is any default StorageClass defined...
If so how can I set up the most simple StorageClass to get this to work, because as I'm self-hosting my openshift origin cluster, I cannot fit into any of the cloud storage option (GCE, AWS, Azure, etc...)?
I think it actually is more related to the setup of storage on your cluster.
There are several storage options as mentioned at OpenShift Origin: Persistent Storage
If you run it locally, you could use NFS (see OpenShift Origin: Persistent Storage using NFS).
If you run it in minishift or single node cluster, you can use HostPath (see Minishift Persistent Volumes). In this case it would be enough to create a PersistentVolume of size 1GB. Then the PersistentVolumeClaim in your template can be bound.
This may be a dumb question but I haven't found much online and want to clarify this.
Given two deployments A and B, both with different container images:
They're deployed in two different pods(different rc, svc etc.) in a K8/OpenShift cluster.
They both need to access the same volume to read files (let's leave locking out of this for now) or at least the same directory structure in that volume.
Mounting this volume using a PVC (Persistent Volume Claim) backed by a PV (Persistent Volume) configured against a NFS share.
Can I confirm that the above would actually be possible? I.e. two different pods connected to the same volume with the same PVC. So they both are reading from the same volume.
Hope that makes sense...
TL;DR
You can share PV and PVC within the same project/namespace for shared volumes (nfs, gluster, etc...), you can also access your shared volume from multiple project/namespaces but it will require project dedicated PV and PVCs, as a PV is bound to single project/namespace and PVC is project/namespace scoped.
Below I've tried to illustrate the current behavior and how PV and PVCs are scoped within OpenShift. These are simple examples using NFS as the persistent storage layer.
the accessModes at this point are just labels, they have no real functionality in terms of controlling access to PV. Below are some examples to show this
the PV is global in the sense that it can be seen/accessed by any project/namespace, HOWEVER once it is bound to a project, it can then only be accessed by containers from the same project/namespace
the PVC is project/namespace specific (so if you have multple projects you would need to have a new PV and PVC for each project to connect to the shared NFS volume - can not reuse the PV from first project)
Example 1:
I have 2 distinct pods running in "default" project/namespace, both accessing the same PV and NFS exported share. Both mount and run fine.
[root#k8dev nfs_error]# oc get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-nfs <none> 1Gi RWO Bound default/nfs-claim 3m
[root#k8dev nfs_error]# oc get pods <--- running from DEFAULT project, no issues connecting to PV
NAME READY STATUS RESTARTS AGE
nfs-bb-pod2-pvc 1/1 Running 0 11m
nfs-bb-pod3-pvc 1/1 Running 0 10m
Example 2:
I have 2 distinct pods running in "default" project/namespace and attempt to create another pod using the same PV but from a new project called testproject to access the same NFS export. The third pod from the new testproject will not be able to bind to the PV as it is already bound by default project.
[root#k8dev nfs_error]# oc get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-nfs <none> 1Gi RWO Bound default/nfs-claim 3m
[root#k8dev nfs_error]# oc get pods <--- running from DEFAULT project, no issues connecting to PV
NAME READY STATUS RESTARTS AGE
nfs-bb-pod2-pvc 1/1 Running 0 11m
nfs-bb-pod3-pvc 1/1 Running 0 10m
** Create a new claim against the existing PV from another project (testproject) and the PVC will fail
[root#k8dev nfs_error]# oc get pvc
NAME LABELS STATUS VOLUME CAPACITY ACCESSMODES AGE
nfs-claim <none> Pending 2s
** nfs-claim will never bind to the pv-nfs PV because it can not see it from it's current project scope
Example 3:
I have 2 distinct pods running in the "default" project and then create another PV and PVC and Pod from testproject. Both projects will be able to access the same NFS exported share but I need a PV and PVC in each of the projects.
[root#k8dev nfs_error]# oc get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-nfs <none> 1Gi RWX Bound default/nfs-claim 14m
pv-nfs2 <none> 1Gi RWX Bound testproject/nfs-claim2 9m
[root#k8dev nfs_error]# oc get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nfs-bb-pod2-pvc 1/1 Running 0 11m
default nfs-bb-pod3-pvc 1/1 Running 0 11m
testproject nfs-bb-pod4-pvc 1/1 Running 0 15s
** notice, I now have three pods running to the same NFS shared volume across two projects, but I needed two PV's as they are bound to a single project, and 2 PVC's, one for each project and the NFS PV I am trying to access
Example 4:
If I by-pass PV and PVC, I can connect to the shared NFS volumes directly from any project using the nfs plugin directly
volumes:
- name: nfsvol
nfs:
path: /opt/data5
server: nfs1.rhs
Now, the volume security is another layer on top of this, using supplementalGroups (for shared storage, i.e. nfs, gluster, etc...), admins and devs should further be able to manage and control access to the shared NFS system.
Hope that helps
I came across this article Learn how to recreate an existing PVC in a new namespace, reusing the same PV with no data losses. I haven't tested it but worth a try. However, k8s docs say PV-to-PVC relationship is one-to-one.
A Note on Namespaces
PersistentVolumes binds are exclusive, and since PersistentVolumeClaims are namespaced objects, mounting claims with "Many" modes (ROX, RWX) is only possible within one namespace.
Reference: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#a-note-on-namespaces
AFAIK, binding a PV multiple times is not supported. You can use volume source (NFS in your case) directly for your use case.