Connection denied while trying to clone data (XtraBackup) from mysql-master (Kubernetes) - mysql

Question

Friends, I trying run a replicated Mysql database with the help of XtraBackup and when it gets to the point when the second node tries to clone data from the master, I'm getting the following error (ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: NO)). This is the point where I get the error:
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql --host=127.0.0.1 --user=${MYSQL_USER} --password=${MYSQL_PASSWORD} --execute="SHOW DATABASES"; do sleep 1; done
echo "Initializing replication from clone position"
mysql -h 127.0.0.1 \
-e "$(<change_master_to.sql.in), \
MASTER_HOST='mysql-0.mysql', \
MASTER_USER='root', \
MASTER_PASSWORD='root-password', \
MASTER_CONNECT_RETRY=10; \
START SLAVE;" || exit 1
mv change_master_to.sql.in change_master_to.sql.orig
fi
Since I am passing a password and the error log it's saying the opposite, I think the problem may be somewhere else but I don't know where. Any Ideas?
Here is my complete yml just in case someone wants to take a look at it:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
serviceName: mysql
replicas: 2
template:
metadata:
labels:
app: mysql
spec:
initContainers:
- name: init-mysql
image: mysql:5.7
command:
- bash
- "-c"
- |
set -ex
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d/
else
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
- name: conf
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
- name: clone-mysql
image: gcr.io/google-samples/xtrabackup:1.0 #bitnami/percona-xtrabackup:8 perconalab/percona-xtrabackup
command:
- bash
- "-c"
- |
set -ex
# Skip the clone if data already exists.
[[ -d /var/lib/mysql/mysql ]] && exit 0
# Skip the clone on master (ordinal index 0).
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal -eq 0 ]] && exit 0
# Clone data from previous peer.
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# Prepare the backup.
xtrabackup --prepare --target-dir=/var/lib/mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_user
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_password
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_root_password
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_database
ports:
- name: mysql
containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 500m
memory: 1Gi
livenessProbe:
exec:
command:
- "bash"
- "-c"
- "mysql --user=${MYSQL_USER} --password=${MYSQL_PASSWORD} --execute=\"SHOW DATABASES;\""
initialDelaySeconds: 40
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- "bash"
- "-c"
- "mysql --user=${MYSQL_USER} --password=${MYSQL_PASSWORD} --execute=\"SHOW DATABASES;\""
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 2
- name: xtrabackup
image: gcr.io/google-samples/xtrabackup:1.0 #perconalab/percona-xtrabackup
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_user
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_password
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: webservice-secrets
key: mysql_root_password
ports:
- name: xtrabackup
containerPort: 3307
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
rm -f xtrabackup_slave_info xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm -f xtrabackup_binlog_info xtrabackup_slave_info
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
fi
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql --host=127.0.0.1 --user='root' --password='root-password' --execute='SHOW DATABASES'; do sleep 1; done
echo "Initializing replication from clone position"
mysql -h 127.0.0.1 \
-e "$(<change_master_to.sql.in), \
MASTER_HOST='mysql-0.mysql', \
MASTER_USER='root', \
MASTER_PASSWORD='root-password', \
MASTER_CONNECT_RETRY=10; \
START SLAVE;" || exit 1
mv change_master_to.sql.in change_master_to.sql.orig
fi
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=root-password"
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 100m
memory: 100Mi
volumes:
- name: conf
emptyDir: {}
- name: config-map
configMap:
name: mysql
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "standard"
resources:
requests:
storage: 1Gi

Answer 1

In below statement or code, you are trying to connect mysql without username and password.
mysql -h 127.0.0.1
-e "$(<change_master_to.sql.in),
MASTER_HOST='mysql-0.mysql',
MASTER_USER='root',
MASTER_PASSWORD='root-password',
MASTER_CONNECT_RETRY=10;
START SLAVE;" || exit 1
Try below once instead of above one so you can say, we need to use username and password to connect local mysql.
mysql --host=127.0.0.1 --user='root' --password='root-password'
-e "$(<change_master_to.sql.in),
MASTER_HOST='mysql-0.mysql',
MASTER_USER='root',
MASTER_PASSWORD='root-password',
MASTER_CONNECT_RETRY=10;
START SLAVE;" || exit 1