Currently I have an InitContainer that creates two directories.
initContainers:
- name: files-init
image: quay.io/quay/busybox
command:
- "/bin/mkdir"
args:
- "-p"
- "/files/test1"
- "/files/test2"
volumeMounts:
- mountPath: /files
name: files
Now it is only creating two directories, but in the future there will be more.
And the readability will be going to be harder and also hard to manage.
Is it possible to use an input file, with for example the name files.txt that has the following lines:
/files/test1
/files/test2
And that this will be read out with something like this?
command:
- "/bin/mkdir"
args:
- "-p $files.txt"
Anyone that can help me with this? Do I need to use ConfigMaps for this, if so how.
Please use configmap to load files into a pod. Below is sample. Also refer K8S documentation for other options as well.
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo
volumes:
- name: foo
configMap:
name: myconfigmap
Related
I'm working on setting up a mysql instance in K8s cluster with TLS support for the client connection.
For that I have setup a cert-manager to issue the self-signed cert. I can see ca.crt, tls.key, tls.crt created in the secrets within my mysql namespace successfully. I followed the following article https://www.jetstack.io/blog/securing-mysql-with-cert-manager/
Now to use this cert, my plan is to place the cert in the /var/lib/mysql directory and update the mysql.conf file using config map. Here is how the mysql.yaml pod spec looks.
kind: Service
metadata:
name: mysql
namespace: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
clusterIP: None
---
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-config
data:
mysql.cnf: |-
[mysqld]
ssl-ca=/var/lib/mysql/ca.crt
ssl-cert= /var/lib/mysql/tls.crt
ssl-key=/var/lib/mysql/tls.key
require_secure_transport=ON
---
apiVersion: v1
kind: Pod
metadata:
name: mysql
labels:
name: mysql
spec:
# securityContext:
# runAsUser: 0
containers:
- image: mysql:5.7
name: mysql
resources: {}
env:
# Use secret in real usage
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-cert-secret
#mountPath: /app/ca.crt
mountPath: /var/lib/mysql/ca.crt
subPath: ca.crt
- name: mysql-cert-secret
mountPath: /var/lib/mysql/tls.crt
#mountPath: /app/tls.crt
subPath: tls.crt
- name: mysql-cert-secret
mountPath: /var/lib/mysql/tls.key
#mountPath: /app/tls.key
subPath: tls.key
- name: config-map-mysqlconf
mountPath: /etc/mysql/mysql.conf
volumes:
- name: mysql-cert-secret
secret:
secretName: mysql-server-tls
- name : config-map-mysqlconf
configMap:
name: mysql-config
If I update the mount path with say /app/ca.crt, then mounting works and I can see the certs in when I access in shell. But for the /var/lib/mysql* I get following error.
Error image
I tried using the securityContext but it didn't help since the directory is accessible by both root and mysql user. Any help would be greatly appreciated. If there is a better way to get this done, I'm happy to try that as well.
This is all done locally using KinD cluster.
Thank you
MySQL stores DB files in /var/lib/mysql by default and there would certainly be an attempt to set the ownership to mysql user. Perhaps here.
Any attempt to update a secret volume will result in an error rather than a successful change as they are read-only projections into the Pod's filesystem. I think that's the reason the article you followed does not suggest anywhere to use dir /var/lib/mysql.
If you still want to attempt this, you can perhaps try by changing the default db storage location to something other than /var/lib/mysql in file /etc/my.cnf or even the default mode of that volumeMount. But I'm not sure if it will work or there won't be any other issues.
I just want to use customized standalone-full.xml in the RHPAM kie server pod which is running in Openshift. I have created the configmap from file and not sure how to set it.
I created the configmap like this
oc create configmap my-config --from-file=standalone-full.xml.
And edited the deploymentconfig of rhpam kie server,
volumeMounts:
- name: config-volume
mountPath: /opt/eap/standalone/configuration
volumes:
- name: config-volume
configMap:
name: my-config
It starts a new container,with sttaus container creating and fails with error(sclaing down 1 to 0)
Am i setting the configmap correct?
You can mount the configmap in a pod as a volume. Here's a good example: just add 'volumes' (to specify configmap as a volume) and 'volumeMounts' (to specify mount point) blocks in the pod's spec:
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "ls /etc/config/" ]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name: special-config
restartPolicy: Never
I am working through the persistent disks tutorial found here while also creating it as a StatefulSet instead of a deployment.
When I run the yaml file into GKE the database fails to start, looking at the logs it has the following error.
[ERROR] --initialize specified but the data directory has files in it. Aborting.
Is it possible to inspect the volume created to see what is in the directory? Otherwise, what am I doing wrong that is causing the disk to be non empty?
Thanks
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: datalayer-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
name: datalayer-svc
labels:
app: myapplication
spec:
ports:
- port: 80
name: dbadmin
clusterIP: None
selector:
app: database
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: datalayer
spec:
selector:
matchLabels:
app: myapplication
serviceName: "datalayer-svc"
replicas: 1
template:
metadata:
labels:
app: myapplication
spec:
terminationGracePeriodSeconds: 10
containers:
- name: database
image: mysql:5.7.22
env:
- name: "MYSQL_ROOT_PASSWORD"
valueFrom:
secretKeyRef:
name: mysql-root-password
key: password
- name: "MYSQL_DATABASE"
value: "appdatabase"
- name: "MYSQL_USER"
value: "app_user"
- name: "MYSQL_PASSWORD"
value: "app_password"
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: datalayer-pv
mountPath: /var/lib/mysql
volumes:
- name: datalayer-pv
persistentVolumeClaim:
claimName: datalayer-pvc
This issue could be caused by the lost+found directory on the filesystem of the PersistentVolume.
I was able to verify this by adding a k8s.gcr.io/busybox container (in PVC set accessModes: [ReadWriteMany], OR comment out the database container):
- name: init
image: "k8s.gcr.io/busybox"
command: ["/bin/sh","-c","ls -l /var/lib/mysql"]
volumeMounts:
- name: database
mountPath: "/var/lib/mysql"
There are a few potential workarounds...
Most preferable is to use a subPath on the volumeMounts object. This uses a subdirectory of the PersistentVolume, which should be empty at creation time, instead of the volume root:
volumeMounts:
- name: database
mountPath: "/var/lib/mysql"
subPath: mysql
Less preferable workarounds include:
Use a one-time container to rm -rf /var/lib/mysql/lost+found (not a great solution, because the directory is managed by the filesystem and is likely to re-appear)
Use mysql:5 image, and add args: ["--ignore-db-dir=lost+found"] to the container (this option was removed in mysql 8)
Use mariadb image instead of mysql
More details might be available at docker-library/mysql issues: #69 and #186
You would usually see if your volumes were mounted with:
kubectl get pods # gets you all the pods on the default namespace
# and
kubectl describe pod <pod-created-by-your-statefulset>
Then you can these commands to check on your PVs and PVCs
kubectl get pv # gets all the PVs on the default namespace
kubectl get pvc # same for PVCs
kubectl describe pv <pv-name>
kubectl describe pvc <pvc-name>
Then you can to the GCP console on disks and see if your disks got created:
You can add the ignore dir lost+found
containers:
- image: mysql:5.7
name: mysql
args:
- "--ignore-db-dir=lost+found"
I use a init container to remove that file (I'm using postgres in my case):
initContainers:
- name: busybox
image: busybox:latest
args: ["rm", "-rf", "/var/lib/postgresql/data/lost+found"]
volumeMounts:
- name: nfsv-db
mountPath: /var/lib/postgresql/data
I am currently running mysql, wordpress and my custom node.js + express application on kubernetes pods in the same cluster. Everything is working quite well but my problem is that all the data will be reset if I have to rerun the deployments, services and persistent volumes.
I have configured wordpress quite extensively and would like to save all the data and insert it again after redeploying everything. How is this possible to do or am I thinking something wrong? I am using the mysql:5.6 and wordpress:4.8-apache images.
I also want to transfer my configuration to my other team members so they don't have to configure wordpress again.
This is my mysql-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: hidden
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
This the wordpress-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
- port: 80
selector:
app: wordpress
tier: frontend
type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:4.8-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
value: hidden
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim
How is this possible to do or am I thinking something wrong?
It might be better to move configuration mindset from working directly on base container instances to configuring container images/manifests. You have several approaches there, just some pointers:
Create own Dockerfile based on images you referenced and bundle configuration files inside them. This is viable approach if configuration is more or less static and can be handled with env vars or infrequent builds of docker images, but require docker registry handling to work with k8s. In this approach you would add all changed files to build context of docker and then COPY them to appropriate places.
Create ConfigMaps and mount them on container filesystem as config files where change is required. This way you can still use base images you reference directly but changes are limited to kubernetes manifests instead of rebuilding docker images. Approach in this case would be to identify all changed files on container, then create kubernetes ConfigMaps out of them and finally mount appropriately. I don't know which exactly things you are changing but here is example of how you can place nginx config in ConfigMap:
kind: ConfigMap
apiVersion: v1
metadata:
name: cm-nginx-example
data:
nginx.conf: |
server {
listen 80;
...
# actual config here
...
}
and then mount it in container in appropriate place like so:
...
containers:
- name: nginx-example
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/nginx/conf.d
name: nginx-conf
volumes:
- name: nginx-conf
configMap:
name: cm-nginx-example
items:
- key: nginx.conf
path: nginx.conf
...
Mount persistent volumes (subpaths) on places where you need configs and keep configuration on persistent volumes.
Personally, I'd probably opt for ConfigMaps since you can easily share/edit those with k8s deployments and configuration details are not lost as some mystical 'extensive work' but can be reviewed, tweaked and stored to some code versioning system for version tracking...
Background:
Currently we're using Docker and Docker Compose for our services. We have externalized the configuration for different environments into files that define environment variables read by the application. For example a prod.env file:
ENV_VAR_ONE=Something Prod
ENV_VAR_TWO=Something else Prod
and a test.env file:
ENV_VAR_ONE=Something Test
ENV_VAR_TWO=Something else Test
Thus we can simply use the prod.env or test.env file when starting the container:
docker run --env-file prod.env <image>
Our application then picks up its configuration based on the environment variables defined in prod.env.
Questions:
Is there a way to provide environment variables from a file in Kubernetes (for example when defining a pod) instead of hardcoding them like this:
apiVersion: v1
kind: Pod
metadata:
labels:
context: docker-k8s-lab
name: mysql-pod
name: mysql-pod
spec:
containers:
-
env:
-
name: MYSQL_USER
value: mysql
-
name: MYSQL_PASSWORD
value: mysql
-
name: MYSQL_DATABASE
value: sample
-
name: MYSQL_ROOT_PASSWORD
value: supersecret
image: "mysql:latest"
name: mysql
ports:
-
containerPort: 3306
If this is not possible, what is the suggested approach?
You can populate a container's environment variables through the use of Secrets or ConfigMaps. Use Secrets when the data you are working with is sensitive (e.g. passwords), and ConfigMaps when it is not.
In your Pod definition specify that the container should pull values from a Secret:
apiVersion: v1
kind: Pod
metadata:
labels:
context: docker-k8s-lab
name: mysql-pod
name: mysql-pod
spec:
containers:
- image: "mysql:latest"
name: mysql
ports:
- containerPort: 3306
envFrom:
- secretRef:
name: mysql-secret
Note that this syntax is only available in Kubernetes 1.6 or later. On an earlier version of Kubernetes you will have to specify each value manually, e.g.:
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_USER
(Note that env take an array as value)
And repeating for every value.
Whichever approach you use, you can now define two different Secrets, one for production and one for dev.
dev-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: mysql-secret
type: Opaque
data:
MYSQL_USER: bXlzcWwK
MYSQL_PASSWORD: bXlzcWwK
MYSQL_DATABASE: c2FtcGxlCg==
MYSQL_ROOT_PASSWORD: c3VwZXJzZWNyZXQK
prod-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: mysql-secret
type: Opaque
data:
MYSQL_USER: am9obgo=
MYSQL_PASSWORD: c2VjdXJlCg==
MYSQL_DATABASE: cHJvZC1kYgo=
MYSQL_ROOT_PASSWORD: cm9vdHkK
And deploy the correct secret to the correct Kubernetes cluster:
kubectl config use-context dev
kubectl create -f dev-secret.yaml
kubectl config use-context prod
kubectl create -f prod-secret.yaml
Now whenever a Pod starts it will populate its environment variables from the values specified in the Secret.
A new update for Kubernetes(v1.6) allows what you asked for(years ago).
You can now use the envFrom like this in your yaml file:
containers:
- name: django
image: image/name
envFrom:
- secretRef:
name: prod-secrets
Where development-secrets is your secret, you can create it by:
kubectl create secret generic prod-secrets --from-env-file=prod/env.txt`
Where the txt file content is a key-value:
DB_USER=username_here
DB_PASSWORD=password_here
The docs are still lakes of examples, I had to search really hard on those places:
Secrets docs, search for --from-file - shows that this option is available.
The equivalent ConfigMap docs shows an example on how to use it.
Note: there's a difference between --from-file and --from-env-file when creating secret as described in the comments below.
When defining a pod for Kubernetes using a YAML file, there's no direct way to specify a different file containing environment variables for a container. The Kubernetes project says they will improve this area in the future (see Kubernetes docs).
In the meantime, I suggest using a provisioning tool and making the pod YAML a template. For example, using Ansible your pod YAML file would look like:
file my-pod.yaml.template:
apiVersion: v1
kind: Pod
...
spec:
containers:
...
env:
- name: MYSQL_ROOT_PASSWORD
value: {{ mysql_root_pasword }}
...
Then your Ansible playbook can specify the variable mysql_root_password somewhere convenient, and substitute it when creating the resource, for example:
file my-playbook.yaml:
- hosts: my_hosts
vars_files:
- my-env-vars-{{ deploy_to }}.yaml
tasks:
- name: create pod YAML from template
template: src=my-pod.yaml.template dst=my-pod.yaml
- name: create pod in Kubernetes
command: kubectl create -f my-pod.yaml
file my-env-vars-prod.yaml:
mysql_root_password: supersecret
file my-env-vars-test.yaml:
mysql_root_password: notsosecret
Now you create the pod resource by running, for example:
ansible-playbook -e deploy=test my-playbook.yaml
This works for me:
file env-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: env-secret
type: Opaque
stringData:
.env: |-
APP_NAME=Laravel
APP_ENV=local
and into the deployment.yaml or pod.yaml
spec:
...
volumeMounts:
- name: foo
mountPath: "/var/www/html/.env"
subPath: .env
volumes:
- name: foo
secret:
secretName: env-secret
````
I smashed my head aupon tyhis for 2 hours now. I found in the docs a very simple solution to minimize my (and hopefully your) pain.
Keep env.prod, env.dev as you have them.
Use a oneliner script to import those into yaml:
kubectl create configmap my-dev-config --from-env-file=env.dev
kubectl create configmap my-prod-config --from-env-file=env.prod
You can see the result (for instant gratification):
# You can also save this to disk
kubectl get configmap my-dev-config -o yaml
As a rubyist, I personally find this solution the DRYest as you have a single point to maintain (the ENV bash file, which is compatible with Python/Ruby libraries, ..) and then you YAMLize it in a single execution.
Note that you need to keep your ENV file clean (I have a lot of comments which prevented this to work so had to prepend a cat config.original | egrep -v "^#" | tee config.cleaned) but this doen't change the complexity substantially.
It's all documented here
This is an old question but let me describe my answer for future beginner.
You can use kustomize configMapGenerator.
configMapGenerator:
- name: example
env: dev.env
and refer this configMap/example in pod definition
This is an old question but it has a lot of viewers so I add my answer.
The best way to separate the configuration from K8s implementation is using Helm. Each Helm package can have a values.yaml file and we can easily use those values in the Helm chart. If we have a multi-component topology we can create an umbrella Helm package and the parent values package also can overwrite the children values files.
You can reference K8S values by specifying them in your container as environment variables.
let your deployment be mongo.yml as follows:
--
kind: Deployment
--
--
containers:
--
env:
- name: DB_URL
valueFrom:
configMapKeyRef:
name: mongo-config
key: mongo-url
- name: MONGO_INITDB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mongo-secret
key: mongo-password
Where mongo-secret is used for senective data, e.g.: passwords or certificates
apiVersion: v1
kind: Secret
metadata:
name: mongo-secret
type: Opaque
data:
mongo-user: bW9uZ291c2Vy
mongo-password: bW9uZ29wYXNzd29yZA==
and mongo-config is used for non-sensitive data
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-config
data:
mongo-url: mongo-service
You could try this steps:
This command will put your file prod.env in the secrets.
kubectl create secret generic env-prod --from-file=prod.env=prod.env
Then, you could reference this file in your deployment.yaml like
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-identity
labels:
app.kubernetes.io/name: api-identity
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: api-identity
template:
metadata:
labels:
app.kubernetes.io/name: api-identity
spec:
imagePullSecrets:
- name: docker-registry-credential
containers:
- name: api-identity
image: "api-identity:test"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8005
protocol: TCP
volumeMounts:
- name: env-file-vol
mountPath: "/var/www/html/.env"
subPath: .env
volumes:
- name: env-file-vol
secret:
secretName: env-prod