Share persistent volume claims amongst containers in Kubernetes/OpenShift - openshift

This may be a dumb question but I haven't found much online and want to clarify this.
Given two deployments A and B, both with different container images:
They're deployed in two different pods(different rc, svc etc.) in a K8/OpenShift cluster.
They both need to access the same volume to read files (let's leave locking out of this for now) or at least the same directory structure in that volume.
Mounting this volume using a PVC (Persistent Volume Claim) backed by a PV (Persistent Volume) configured against a NFS share.
Can I confirm that the above would actually be possible? I.e. two different pods connected to the same volume with the same PVC. So they both are reading from the same volume.
Hope that makes sense...

TL;DR
You can share PV and PVC within the same project/namespace for shared volumes (nfs, gluster, etc...), you can also access your shared volume from multiple project/namespaces but it will require project dedicated PV and PVCs, as a PV is bound to single project/namespace and PVC is project/namespace scoped.
Below I've tried to illustrate the current behavior and how PV and PVCs are scoped within OpenShift. These are simple examples using NFS as the persistent storage layer.
the accessModes at this point are just labels, they have no real functionality in terms of controlling access to PV. Below are some examples to show this
the PV is global in the sense that it can be seen/accessed by any project/namespace, HOWEVER once it is bound to a project, it can then only be accessed by containers from the same project/namespace
the PVC is project/namespace specific (so if you have multple projects you would need to have a new PV and PVC for each project to connect to the shared NFS volume - can not reuse the PV from first project)
Example 1:
I have 2 distinct pods running in "default" project/namespace, both accessing the same PV and NFS exported share. Both mount and run fine.
[root#k8dev nfs_error]# oc get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-nfs <none> 1Gi RWO Bound default/nfs-claim 3m
[root#k8dev nfs_error]# oc get pods <--- running from DEFAULT project, no issues connecting to PV
NAME READY STATUS RESTARTS AGE
nfs-bb-pod2-pvc 1/1 Running 0 11m
nfs-bb-pod3-pvc 1/1 Running 0 10m
Example 2:
I have 2 distinct pods running in "default" project/namespace and attempt to create another pod using the same PV but from a new project called testproject to access the same NFS export. The third pod from the new testproject will not be able to bind to the PV as it is already bound by default project.
[root#k8dev nfs_error]# oc get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-nfs <none> 1Gi RWO Bound default/nfs-claim 3m
[root#k8dev nfs_error]# oc get pods <--- running from DEFAULT project, no issues connecting to PV
NAME READY STATUS RESTARTS AGE
nfs-bb-pod2-pvc 1/1 Running 0 11m
nfs-bb-pod3-pvc 1/1 Running 0 10m
** Create a new claim against the existing PV from another project (testproject) and the PVC will fail
[root#k8dev nfs_error]# oc get pvc
NAME LABELS STATUS VOLUME CAPACITY ACCESSMODES AGE
nfs-claim <none> Pending 2s
** nfs-claim will never bind to the pv-nfs PV because it can not see it from it's current project scope
Example 3:
I have 2 distinct pods running in the "default" project and then create another PV and PVC and Pod from testproject. Both projects will be able to access the same NFS exported share but I need a PV and PVC in each of the projects.
[root#k8dev nfs_error]# oc get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv-nfs <none> 1Gi RWX Bound default/nfs-claim 14m
pv-nfs2 <none> 1Gi RWX Bound testproject/nfs-claim2 9m
[root#k8dev nfs_error]# oc get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nfs-bb-pod2-pvc 1/1 Running 0 11m
default nfs-bb-pod3-pvc 1/1 Running 0 11m
testproject nfs-bb-pod4-pvc 1/1 Running 0 15s
** notice, I now have three pods running to the same NFS shared volume across two projects, but I needed two PV's as they are bound to a single project, and 2 PVC's, one for each project and the NFS PV I am trying to access
Example 4:
If I by-pass PV and PVC, I can connect to the shared NFS volumes directly from any project using the nfs plugin directly
volumes:
- name: nfsvol
nfs:
path: /opt/data5
server: nfs1.rhs
Now, the volume security is another layer on top of this, using supplementalGroups (for shared storage, i.e. nfs, gluster, etc...), admins and devs should further be able to manage and control access to the shared NFS system.
Hope that helps

I came across this article Learn how to recreate an existing PVC in a new namespace, reusing the same PV with no data losses. I haven't tested it but worth a try. However, k8s docs say PV-to-PVC relationship is one-to-one.
A Note on Namespaces
PersistentVolumes binds are exclusive, and since PersistentVolumeClaims are namespaced objects, mounting claims with "Many" modes (ROX, RWX) is only possible within one namespace.
Reference: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#a-note-on-namespaces

AFAIK, binding a PV multiple times is not supported. You can use volume source (NFS in your case) directly for your use case.

Related

error when creating ".": persistentvolumeclaims "wp-pv-claim" is forbidden: exceeded quota

I'm trying to run WordPress by using Kubernetes link, and the only change is I changed 20Gi to 5Gi, but when I run kubectl apply -k ., I get this error:
Error from server (Forbidden): error when creating ".": persistentvolumeclaims "wp-pv-claim" is forbidden: exceeded quota: storagequota, requested: requests.storage=5Gi, used: requests.storage=5Gi, limited: requests.storage=5Gi
I searched but did not find any related answer to mine (or even maybe I'm wrong).
Could you please answer me these questions:
How to solve the above issue?
If the volume's size is limited to 5G, then the pod cannot be bigger than 5G? I mean if I exec into the pod and run a command like dd if=/dev/zero of=file bs=1M count=8000, should it create an 8G file or not? I mean this quota and volume limits whole the pod? Or only a specific path like /var/www/html?
Edit 1
describe pvc mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: app=wordpress
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: wordpress-mysql-6c479567b-vzpm5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 4m (x222 over 59m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
I decided to summarize our comments conversation for better readability and visibility.
The issue at first seemed to be caused by resourcequota.
Error from server (Forbidden): error when creating ".": persistentvolumeclaims "wp-pv-claim" is forbidden: exceeded quota: storagequota, requested: requests.storage=5Gi, used: requests.storage=5Gi, limited: requests.storage=5Gi
It looked like there was already existing PVC and it wouldn't allow to create a new one.
OP removed the resource quota although it was not necessary in this case since the real issue was with the PVC.
kubectl describe pvc mysql-pv-claim showed the following event:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 4m (x222 over 59m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Event message:
persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Since OP created the cluster with kubeadm and kubeadm doesn't come with a predeployed storage provider out of the box; this means that it needs to be added manually. (Storage Provider is a controller that can create a volume and mount it).
Each StorageClass has a provisioner that determines what volume plugin is used for provisioning PVs. This field must be specified. Since there was no storage class in cluster, OP decided to create one and picked Local storage class but forgot that:
Local volumes do not currently support dynamic provisioning [...].
and
Local volumes can only be used as a statically created PersistentVolume. Dynamic provisioning is not supported
This means that a local volume had to be created manually.

Pod level route restriction

EDITED:
I have a service running in OpenShift on 2 pods, let's call them P1 and P2.
The service does two things:
An API
We listen to Kafka messages from a topic and then process them.
Is there a way I can restrict all calls made to API only to P1 and all calls for Kafka only to P2 ?
My suggestion may not fit with your requests, but if each one pod is running in a specific project, then it would be available as follows.
First, you should configure pod's source IP statically using Egress IP based on project level, refer Enabling Static IPs for External Project Traffic for more details.
$ oc patch netnamespace p1_project -p '{"egressIPs": ["1.1.1.1"]}'
$ oc patch netnamespace p2_project -p '{"egressIPs": ["2.2.2.2"]}'
After that, you can allow each pod IP based on whitelist, refer Route-specific IP Whitelists for more details.
kind: Route
metadata:
name: R1
annotations:
haproxy.router.openshift.io/ip_whitelist: 1.1.1.1
kind: Route
metadata:
name: R2
annotations:
haproxy.router.openshift.io/ip_whitelist: 2.2.2.2
I hope it help you.

Kubernetes -- Helm -- Mysql Chart loses stored data after stopping pod

Using https://github.com/helm/charts/tree/master/stable/mysql (all the code is here), it is cool being able to run mysql as part of my local kubernetes cluster (using docker kubernetes).
The problem though is that once I stop running the pod, and then run the pod again, all the data that was stored is now gone.
My question is how do I keep the data that was added to the mysql pod? I have read about persistent volumes, and the mysql helm example from github is showing that it is using PersistentVolumeClaim. I have also enabled persistence on the values.yaml file, but I cannot seem to have the same data that was saved in the database.
My docker kubernetes version is currently 1.14.6.
Please verify your msql POD You should notice volumes and volumesMount options:
volumeMounts:
- mountPath: /var/lib/mysql
name: data
.
.
.
volumes:
- name: data
persistentVolumeClaim:
claimName: msq-mysql
In additions please verify your PersistentVolume and PersistentVolumeClaim, storageClass:
kubectl get pv,pvc,pods,sc:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-2c6aa172-effd-11e9-beeb-42010a840083 8Gi RWO Delete Bound default/msq-mysql standard 24m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/msq-mysql Bound pvc-2c6aa172-effd-11e9-beeb-42010a840083 8Gi RWO standard 24m
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/msq-mysql-b5c48c888-pz6p2 1/1 Running 0 4m28s 10.0.0.8 gke-te-1-default-pool-36546f4e-5rgw <none> <none>
Please run kubectl describe persistentvolumeclaim/msq-mysql (in your example you should change the pvc name)
You can notice that pvc was provisioned successfully using gce-pd and mounted by msq-mysql POD.
Normal ProvisioningSucceeded 26m persistentvolume-controller Successfully provisioned volume pvc-2c6aa172-effd-11e9-beeb-42010a840083 using kubernetes.io/gce-pd
Mounted By: msq-mysql-b5c48c888-pz6p2
I have created table with on row, deleted the pod and verified after that (as expected everything is alright):
mysql> SELECT * FROM t;
+------+
| c |
+------+
| ala |
+------+
1 row in set (0.00 sec)
Why: all the data that was stored is now gone.
As per helm chart docs:
The MySQL image stores the MySQL data and configurations at the /var/lib/mysql path of the container.
By default a PersistentVolumeClaim is created and mounted into that directory. In order to disable this functionality you can change the values.yaml to disable persistence and use an emptyDir instead.
Mostly there is problem with pv,pvc binding. It can be also problem with user defined or non default storageClass.
So please verify pv,pvc as stated above.
Take a look at StorageClass
A claim can request a particular class by specifying the name of a StorageClass using the attribute storageClassName. Only PVs of the requested class, ones with the same storageClassName as the PVC, can be bound to the PVC.
PVCs don’t necessarily have to request a class. A PVC with its storageClassName set equal to "" is always interpreted to be requesting a PV with no class, so it can only be bound to PVs with no class (no annotation or one set equal to ""). A PVC with no storageClassName is not quite the same and is treated differently by the cluster, depending on whether the DefaultStorageClass admission plugin is turned on.

How to deploy OpenShift Origin 10 (OKD) on one node with GlusterFS

I am able to install OKD on one node and scaleup on multiple node accordngly.
But now i want to install OKD with GlusterFS on one node and then extend this on multiple nodes.
Currently i am getting error that at least three nodes required. How i can bypass this check in ansible?
As per github documentations i have three options
Configuring a new, natively-hosted GlusterFS cluster. In this scenario, GlusterFS pods are deployed on nodes in the OpenShift cluster which are configured to provide storage.
Configuring a new, external GlusterFS cluster. In this scenario, the cluster nodes have the GlusterFS software pre-installed but have not been configured yet. The installer will take care of configuring the cluster(s) for use by OpenShift applications.
Using existing GlusterFS clusters. In this scenario, one or more GlusterFS clusters are assumed to be already setup. These clusters can be either natively-hosted or external, but must be managed by a heketi service.
Can option 2 or 3 be used to start with one node and extend accordingly? I have install glusterfs cluster on one node and extend it to second node but how to introduce in openshift?
https://imranrazakh.blogspot.com/2018/08/
I found one way to install glusterfs on one node, Find below all in one installation with glusterfs
Changed inventory file like below
[OSEv3:children]
masters
nodes
etcd
glusterfs
[OSEv3:vars]
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
ansible_ssh_user=root
openshift_deployment_type=origin
openshift_enable_origin_repo=false
openshift_disable_check=disk_availability,memory_availability
os_firewall_use_firewalld=true
openshift_public_hostname=console.1.1.0.1.nip.io
openshift_master_default_subdomain=apps.1.1.0.1.nip.io
openshift_storage_glusterfs_is_native=false
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_heketi_is_native=true
openshift_storage_glusterfs_heketi_executor=ssh
openshift_storage_glusterfs_heketi_ssh_port=22
openshift_storage_glusterfs_heketi_ssh_user=root
openshift_storage_glusterfs_heketi_ssh_sudo=false
openshift_storage_glusterfs_heketi_ssh_keyfile="/root/.ssh/id_rsa
[masters]
1.1.0.1 openshift_ip=1.1.0.1 openshift_schedulable=true
[etcd]
1.1.0.1 openshift_ip=1.1.0.1
[nodes]
1.1.0.1 openshift_ip=1.1.0.1 openshift_node_group_name="node-config-all-in-one" openshift_schedulable=true
[glusterfs]
1.1.0.1 glusterfs_devices='[ "/dev/vdb" ]'
Now we have to hack ansible script as it expect three nodes by adding --durability none in following ansible script
openshift-ansible/roles/openshift_storage_glusterfs/tasks/heketi_init_db.yml
Following is updated snippet
- name: Create heketi DB volume
command: "{{ glusterfs_heketi_client }} setup-openshift-heketi-storage --image {{ glusterfs_heketi_image }} --listfile /tmp/heketi-storage.json --durability none"
register: setup_storage
As by default it create StorageClass which expect replicate environment, so we have to create custom storageclass like below with "volumetype: none"
oc create -f - <<EOT
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: glusterfs-nr-storage
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
parameters:
resturl: http://heketi-storage.glusterfs.svc:8080
restuser: admin
secretName: heketi-storage-admin-secret
secretNamespace: glusterfs
volumetype: none
provisioner: kubernetes.io/glusterfs
volumeBindingMode: Immediate
EOT
Now you can create storage dynamically from webconsole :) Any suggestions for improvement are welcome.
Next i will check how i can extend it?

How to compare memory quota control implementation, openshift vs. docker

My customer asked me if openshift can provide the same control on memory usage as docker can, for example, docker run can have the following parameters to control memory usage when running a container:
--kernel-memory
--memory
--memory-reservation
While I searched the corresponding part in openshift, I found ResoureQuota and LimitRange should work for that, but what if a pod claims itself will use 100Mi memory by using LimitRange but actually it will consume 500Mi memory instead? the memory can still be used "illegally", seems docker with --memory can control this situation more better.
In openshift, is there any method for controlling real memory usage instead of checking what a pod claimed in LimitRange or using "oc set resources dc hello --requests=memory=256Mi"?
Best regards
Lan
As far as my experience with Openshift I have not come across the situation where the POD has consumed more memory or CPU for which it has configured. If in case it reaches the threshold, the POD automatically will be killed and restarts.
You can set the POD resource limits in the Deployment config:
resources:
limits:
cpu: 750m
memory: 1024Mi
The resources can be monitored in the metrics section of the respective POD:
Apart from the indiviual POD settings you can define your own overall project settings for each container in the POD.
$ oc get limits
NAME
limits
$ oc describe limits <NAME>
Name: <NAME>
Namespace: <NAME_SPACE>
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod memory 256Mi 32Gi - - -
Pod cpu 125m 6400m - - -
Container cpu 125m 6400m 125m 750m -
Container memory 256Mi 32Gi 512Mi 1Gi -
For more information on resource settings refer here.
If you only use --requests=memory=256Mi, you set QoS level to "burstable", which means pod can request at least 256Mi memory without upper limit except reaching project quota. If you want to limit pod memory, use --limit=memory=256Mi instead.