MySQL ram usages on default pod - mysql

Im trying to run a basic MySQL 8 pod in Kubernetes. I did a basic deployment without any resource limits or whatsoever. What i do notice that the memory consumption is high. I have a nearly empty database (i think there are max 100 rows with basic data) and the pod is consuming 750M memory.
Is there anything that you can do about this?
apiVersion: apps/v1
kind: Deployment
metadata:
name: db
namespace: my-namespace
spec:
replicas: 1
selector:
matchLabels:
app: db
strategy:
type: Recreate
template:
metadata:
labels:
app: db
spec:
containers:
- name: db
image: mysql:8.0
resources:
env:
- name: MYSQL_DATABASE
value: mydb
- name: MYSQL_USER
value: myuser
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: secret
key: "DATABASE_PASSWORD"
ports:
- containerPort: 3306
name: transport
protocol: TCP
volumeMounts:
- name: db
mountPath: /var/lib/mysql
subPath: mysql
volumes:
- name: db
persistentVolumeClaim:
claimName: db
Best
Pim

I've just ran a docker container with MySQL and it consumes ~400M of RAM with an empty database that is not even being queried by some application.
docker run -itd -e MYSQL_ROOT_PASSWORD=password mysql
docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
a779ef705921 happy_lumiere 2.22% 373.4MiB / 1.943GiB 18.77% 1.05kB / 0B 0B / 0B 38
According to this assessment it seems to me that 750M is not that much.
You can check out these Best Practices for Configuring Optimal MySQL Memory Usage.

In the future please add more information about your environment (local, kubeadm, minikube, cloud) and scenario. It would be easier to reproduce or troubleshoot.
I would not say that 750M RAM is high consumption. For test I've deployed MySQL 8.0 on my GKE cluster using HELM based on this chart. I've only changed default MySql image version to 8.0.
$ helm install sql stable/mysql
Pure instance of MySQL on my cluster without any limits, requests or data:
$ kubectl top pods
NAME CPU(cores) MEMORY(bytes)
sql-mysql-6c9489d5b9-m8zmh 8m 376Mi
So if MySQL works with actual data it's normal to consume more resources.
I would say it's normal behaviour.
For general resources use in MySql you can check How MySQL Uses Memory.
However if you working on local environment with some limited resources you should specify Limits in your YAMLs.

What i did to reduce the consumption mainly is disable the performance schema. Further i reduced the default connection limit to ensure that it does not reserve the memory for those who are not being used.
It will indeed mean that it can collapse if there are more users but i guess its a guess in on one of those
more cluster ram or failing to run it due to ram exhaustion
hit the max connection limit
Proper monitoring on all will do the trick

Related

GKE Kubernetes MySQL Input/output error Ext4Error

I have deployed a MySQL database (statefulset) on Kubernetes zonal cluster, running as a service (GKE) in Google Cloud Platform.
The zonal cluster consist of 3 instances of type e2-medium.
The MySQL container cannot start due to the following error.
kubectl logs mysql-statefulset-0
2022-02-07 05:55:38+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.35-1debian10 started.
find: '/var/lib/mysql/': Input/output error
Last seen events.
4m57s Warning Ext4Error gke-cluster-default-pool-rnfh kernel-monitor, gke-cluster-default-pool-rnfh EXT4-fs error (device sdb): __ext4_find_entry:1532: inode #2: comm mysqld: reading directory lblock 0 40d 8062 gke-cluster-default-pool-rnfh
3m22s Warning BackOff pod/mysql-statefulset-0 spec.containers{mysql} kubelet, gke-cluster-default-pool-rnfh Back-off restarting failed container
Nodes.
kubectl get node -owide
gke-cluster-default-pool-ayqo Ready <none> 54d v1.21.5-gke.1302 So.Me.I.P So.Me.I.P Container-Optimized OS from Google 5.4.144+ containerd://1.4.8
gke-cluster-default-pool-rnfh Ready <none> 54d v1.21.5-gke.1302 So.Me.I.P So.Me.I.P Container-Optimized OS from Google 5.4.144+ containerd://1.4.8
gke-cluster-default-pool-sc3p Ready <none> 54d v1.21.5-gke.1302 So.Me.I.P So.Me.I.P Container-Optimized OS from Google 5.4.144+ containerd://1.4.8
I also noticed that rnfh node is out of memory.
kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-cluster-default-pool-ayqo 117m 12% 992Mi 35%
gke-cluster-default-pool-rnfh 180m 19% 2953Mi 104%
gke-cluster-default-pool-sc3p 179m 19% 1488Mi 52%
MySql mainfest
# HEADLESS SERVICE
apiVersion: v1
kind: Service
metadata:
name: mysql-headless-service
labels:
kind: mysql-headless-service
spec:
clusterIP: None
selector:
tier: mysql-db
ports:
- name: 'mysql-http'
protocol: 'TCP'
port: 3306
---
# STATEFUL SET
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql-statefulset
spec:
selector:
matchLabels:
tier: mysql-db
serviceName: mysql-statefulset
replicas: 1
template:
metadata:
labels:
tier: mysql-db
spec:
terminationGracePeriodSeconds: 10
containers:
- name: my-mysql
image: my-mysql:latest
imagePullPolicy: Always
args:
- "--ignore-db-dir=lost+found"
ports:
- name: 'http'
protocol: 'TCP'
containerPort: 3306
volumeMounts:
- name: mysql-pvc
mountPath: /var/lib/mysql
env:
- name: MYSQL_ROOT_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: mysql-root-username
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: mysql-root-password
- name: MYSQL_USER
valueFrom:
configMapKeyRef:
name: mysql-config
key: mysql-username
- name: MYSQL_PASSWORD
valueFrom:
configMapKeyRef:
name: mysql-config
key: mysql-password
- name: MYSQL_DATABASE
valueFrom:
configMapKeyRef:
name: mysql-config
key: mysql-database
volumeClaimTemplates:
- metadata:
name: mysql-pvc
spec:
storageClassName: 'mysql-fast'
resources:
requests:
storage: 120Gi
accessModes:
- ReadWriteOnce
- ReadOnlyMany
MySQL storage class manifest:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mysql-fast
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: Immediate
Why Kubernetes is trying to schedule pod in out of memory node?
UPDATES
I've added requests and limits to MySQL manifest to improve the Qos Class. Now the Qos Class is Guaranteed.
Unfortunately, Kubernetes still trying to schedule to out of memory rnfh node.
kubectl describe po mysql-statefulset-0 | grep node -i
Node: gke-cluster-default-pool-rnfh/So.Me.I.P
kubectl describe po mysql-statefulset-0 | grep qos -i
QoS Class: Guaranteed
I ran a few more tests but I couldn't replicate this.
To answer this one correctly, we would need much more logs. Not sure if you still have them. If I could guess which was the root cause of this issue I would say it was connected with the PersistentVolume.
In one of the Github issue - Volume was remounted as read only after error #752 I found very similar behavior to OP's behavior.
You have created a special storageclass for your MySQL. You've set reclaimPolicy: Retain so PV was not removed. When Statefulset pod (with the same suffix -0) has been recreated (restarted due to error with connectivity, some issues on DB, hard to say) it tried to re-claim this Volume. In the mentioned Github issue, user had very similar situation. Also got inode #262147: comm mysqld: reading directory lblock issue, but in the bellow there was also entry [ +0.003695] EXT4-fs (sda): Remounting filesystem read-only. Maybe it changed permissions when re-mounted?
Another thing that your volumeClaimTemplates contained
accessModes:
- ReadWriteOnce
- ReadOnlyMany
So one PersistentVolume could be used as ReadWriteOnce by one node or only ReadOnlyMany by many nodes. There is a possibility that POD was recreated in different node with Read-Only assessMode.
[ +35.912075] EXT4-fs warning (device sda): htree_dirblock_to_tree:977: inode #2: lblock 0: comm mysqld: error -5 reading directory block
[ +6.294232] EXT4-fs error (device sda): ext4_find_entry:1436: inode #262147: comm mysqld: reading directory lblock ...
[ +0.005226] EXT4-fs error (device sda): ext4_find_entry:1436: inode #2: comm mysqld: reading directory lblock 0
[ +1.666039] EXT4-fs error (device sda): ext4_journal_check_start:61: Detected aborted journal
[ +0.003695] EXT4-fs (sda): Remounting filesystem read-only
It would fit to OP's comment:
Two days ago for reasons unknown to me Kubernetes restarted the container and was keep trying to run it on rnfa machine. The container was probably evicted from another node.
Another thing is that node or cluster might be updated (depending if the auto update option was turned on) which might enforce restart of the pod.
Issue with '/var/lib/mysql/': Input/output error might point to database corruption like mentioned here.
In general, the issue has been resolved by cordoning affected node. Additional information about the difference between cordon and drain can be found here.
Just as an addition, to assign pods to specific node or node with specified label, you can use Affinity

MySQL Dump Importing Speed Super Slow for my Kuberenetes MySQL Pod

I am migrating my MySQL database from bare metal setup to Kubernetes. So I have exported a MySQL dump with size of around 8.9GB and uploaded the MySQL dump on my Kubernetes master node. The dump is inserted using the command
kubectl exec -it [podname] -n [namespace] -- mysql -u [db user] -p[password] [db name] < [name of the dump].sql
The insert speed is super slow and thus I import the tables one by one to observe its behavior. A 1.8GB dump takes more than 5 hours to complete.
The SELECT command itself take 0.013 seconds to select 1000 entries. The INSERT INTO for batch of data can take up to 72 seconds to 120 seconds.
I search through the internet and find that the MySQL dump insertion speed is slow to container.
Does anyone experience the same? And can give me some clue to speed up the import speed of the dump?
Some details of my cluster
MySQL Pod version: MySQL version 5.7
Kubernetes Version: v1.20.9
File System: btrfs
The MySQL Pod is deployed using pod deployment and the db is dumped to the pvc defined.
YAML config of the Pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysqldb01
spec:
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
replicas: 1
selector:
matchLabels:
app: mysqldb01
template:
metadata:
labels:
app: mysqldb01
spec:
schedulerName: stork
containers:
- name: mysql
image: mysql:5.7
imagePullPolicy: "Always"
env:
- name: MYSQL_ROOT_PASSWORD
value: xxxxxxxxxxxxxxxxxxxxxxxxx
args:
- --lower_case_table_names=1
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql-data
- name: mysql-custom-config
mountPath: /etc/mysql/mysql.conf.d/custom.my.cnf
subPath: my.custom.conf
volumes:
- name: mysql-data
persistentVolumeClaim:
claimName: px-mysql-db01-pvc
- name: mysql-custom-config
configMap:
name: mysql-custom-config
One master node and three workers node are installed with rancher.
Kubernetes is installed using
curl https://releases.rancher.com/install-docker/20.10.sh | sh
Thanks in advance.

statefulset unable to rollback if the pods are not in running state

I have deployed mongo stateful pods with an auto rolling strategy and below is the template for it. The deployment is successful and the pods are into Running state.
- apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
podManagementPolicy: Parallel
replicas: 3
strategy:
type: Rolling
template:
metadata:
labels:
role: mongo
environment: test
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo:4.0
imagePullPolicy: Always
command:
- mongod
- "--replSet"
- rs0
- "--bind_ip"
- 0.0.0.0
- "--smallfiles"
- "--noprealloc"
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
updateStrategy:
type: RollingUpdate
I am trying to update the image of the mongo using the following set command,
oc set image statefulset/mongo mongo=mongo:4.2 -n mongo-replica
While trying to update the image, the pods are into "CrashLoopBackOff" error. I am expecting the pods to be auto rolled back to the previous running version.
But the pods are struck in "CrashLoopBackOff" error state. I want the pods to be rolled back to the previous running version. Any suggestions here would be appreciated.
Statefulset unfortunately don't have a Rollback, but you can warranty your services using the probes, having a well configure Liveness and Readiness probes the changed version will only take the place of the running version with the probes answering an ok status.
In that way only one of your 3 replicas will crash in a failure, and you can work on it to solve the problem or manually rollback your changes, but without losing the delivery of your service.
More detail about this you can see on the k8s documentation:
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#forced-rollback
About the probes, you can get a good explanation about it here:
https://www.openshift.com/blog/liveness-and-readiness-probes

MySql server deployed in kubernetes is very slow to respond

I have deployed mysql in kubernetes. The pods are up and running. But when I tried to create db, table and insert data, all these operations seems to be very slow. Here is the yaml files I used for deployment. Can you look into the yaml and suggest me what could be the reason for it.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
serviceName: "mysql"
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:8.0.20
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumeClaimTemplates:
- metadata:
name: mysql-persistent-storage
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: rbd-default
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
type: NodePort
ports:
- port: 3306
targetPort: 3306
selector:
app: mysql
I tried creating database after I exec into the pod, the operation took 40 sec to complete. When I tried connecting it to visual studio and perform same operation it took me more than 4 minutes. I think 40 sec itself is too long. However fetching data just took 300 ms from visual studio.
I connected it to visual studio using IP and node port
Thank you all for spending time to answer the question. I think I solved the issue. It was basically the storage class that I used which was causing the issue. Once I updated it to rbd-fast, the response got much faster.
This sounds like a DNS lookup delay/timeout.
If so, you need to set skip_name_resolve in your config file, in the [mysqld] section:
https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_skip_name_resolve
Note that you will not be able to use 'user'#'host' for access permissions if you do this, you will have to 'use user'#'ip_address' in your access permissions. You can still use the % wildcard, e.g. 'user'#'10.0.%' is valid. 'user'#'localhost' will also continue to work as this actually means local socket.

Unable to run mysql pod in kubernetes with external volume

I have google cloud container engine setup. I wanted to spin pod of mysql with external volume.
ReplicationController:
apiVersion: v1
kind: ReplicationController
metadata:
labels:
name: mysql
name: mysql-controller
spec:
replicas: 1
template:
metadata:
labels:
name: mysql
spec:
containers:
- image: mysql
name: mysql
ports:
- name: mysql
containerPort: 3306
hostPort: 3306
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
gcePersistentDisk:
pdName: mysql-1-disk
fsType: ext4
When i run RC without external volume, MySQL works fine. It breaks with below error when i try to attach volume
Kubernetes POD Error:
Warning FailedSyncError syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 20s restarting failed container=mysql pod=mysql-controller-4hhqs_default(eb34ff46-8784-11e6-8f12-42010af00162)"
Disk (External Volume):
mysql-1-disk is the google cloud disk. I tried creating disk with both blank disk and image - ubuntu. Both failed with same error.
The error messages on mounting persistent disks are really not descriptive from my perspective. Use a blank disk based on your configuration file.
Some things to check:
Is the pdName exactly the same as in your CGE environment
Is the disk in the same availability zone (eg. europe-west1-c) as your cluster, otherwise it can't mount.
Hope this helps.
The problem that you face may be caused by using RC, not Pod to interact with the Persistent Disk.
As it's mentioned in documentation:
A feature of PD is that they can be mounted as read-only by multiple consumers simultaneously. This means that you can pre-populate a PD with your dataset and then serve it in parallel from as many pods as you need. Unfortunately, PDs can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed.
Using a PD on a pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1.
In this case, I may suggest you to run MySQL with Persistent Disks defining the disk connection in Pod configuration file. Sample configuration you may find here.