Can't run a MySQL slave instance on GKE - mysql

Question

I have a MySQL service running on Google Kubernetes Engine configured with Master-Slave replication strategy. Unfortunately up until now only I am only managed to have the master (mysql-0) instance running. Slave instances (mysql-n) always stuck on CrashLoopBackOff.
Current pods:
When I check the log for mysql-1 instance, the following lines printed:
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08 08:11:20+00:00 [Note] [Entrypoint]: Initializing database files
2020-01-08T08:11:20.787668Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-08T08:11:20.787780Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.18) initializing of server in progress as process 45
2020-01-08T08:11:20.789731Z 0 [ERROR] [MY-010457] [Server] --initialize specified but the data directory has files in it. Aborting.
2020-01-08T08:11:20.789739Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2020-01-08T08:11:20.790259Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-01-08T08:11:20.791368Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.18) MySQL Community Server - GPL.
On the master MySQL Instance (mysql-0), the following lines printed:
2020-01-08 08:08:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08 08:08:03+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-08 08:08:03+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-08T08:08:03.605134Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-08T08:08:03.605280Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18) starting as process 1
2020-01-08T08:08:05.939722Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-08T08:08:05.945787Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-08T08:08:06.069048Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.18' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
2020-01-08T08:08:06.303012Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060
The key difference between the two logs is, on the slave instance, the line Initializing database files are printed, indicating mysqld attempted to initalize already existing mysql data directory (/var/lib/mysql) whereas it doesn't happen when the master instance initializes.
I was following this kubernetes guide but it doesn't work on MySQL version 8. So I modify the StatefulSet deployment. Here is the yaml snippet
Anybody has idea on how to solve this problem? Much appreciated :)
UPDATE:
Solution provided by #BinaryBullet actually works with some changes down the line to enable multiple slaves replication. This yaml file is the end result.

Answer 1

I could see this is the issue with your volumemount, as you have single VolumeClaimTemplate for both master and slave mysql-bin files, schemas, ib_buffer_pool, etc all other mysql related files are conflicting with each other and trying to save at the same location. Due to this one of your db instance is failing, i.e. which ever starting later is failing, in your case slave db.
This could be avoided by adding separate PV and PVC for your master and slave.
Addition of something like this to your statefulset
volumeClaimTemplates:
- metadata:
name: slave-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
And volumemount for slave pod is something like this
volumeMounts:
- name: slave-data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
But make sure you have a separate Persistent Volume for your slave as well. To which all your slave mysql-bin files, ib_buffer_pool, schemas, etc mysql related files get saved in a different location from master.
Hope this helps.