I'm sure there is a simple fix for this, but forgive me I'm new at PXC. I'm using rsync to transfer state of the bootstrapping node to node2. node2 is the node I am joining to the cluster. I originally tried Xtrabackup, but ran into problems which I will explore at another time. For now I'm using rsync for what I thought would be simplicity. If you scroll down to the [ERROR] you will see where the problems are causing the State Transfer to be interrupted. What could be causing this?
*WSREP_SST: **[ERROR]** find/rsync returned code 123: (20141228 02:24:40.505)
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Failed to read from: **wsrep_sst_rsync** --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket*
2014-12-28 02:24:24 9446 [Note] WSREP: (2861d1d7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2014-12-28 02:24:24 9446 [Note] WSREP: declaring 41f79045 at tcp://192.168.70.2:4567 stable
2014-12-28 02:24:24 9446 [Note] WSREP: Node 2861d1d7 state prim
2014-12-28 02:24:24 9446 [Note] WSREP: save pc into disk
2014-12-28 02:24:24 9446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2014-12-28 02:24:24 9446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 42495aba-8e30-11e4-9596-c702faf22ad0
2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: sent state msg: 42495aba-8e30-11e4-9596-c702faf22ad0
2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 42495aba-8e30-11e4-9596-c702faf22ad0 from 0 (node3)
2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 42495aba-8e30-11e4-9596-c702faf22ad0 from 1 (node2)
2014-12-28 02:24:24 9446 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 1,
members = 1/2 (joined/total),
act_id = 0,
last_appl. = 0,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = 48ec9889-8ddc-11e4-9efd-da6610fd24da
2014-12-28 02:24:24 9446 [Note] WSREP: Flow-control interval: [23, 23]
2014-12-28 02:24:24 9446 [Note] WSREP: New cluster view: global state: 48ec9889-8ddc-11e4-9efd-da6610fd24da:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
2014-12-28 02:24:24 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:24 9446 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-28 02:24:24 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:24 9446 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2014-12-28 02:24:24 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:25 9446 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node3)(SYNCED) as donor.
2014-12-28 02:24:25 9446 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
2014-12-28 02:24:25 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:25 9446 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0''
2014-12-28 02:24:25 9446 [Note] WSREP: sst_donor_thread signaled with 0
2014-12-28 02:24:25 9446 [Note] WSREP: Flushing tables for SST...
2014-12-28 02:24:25 9446 [Note] WSREP: Provider paused at 48ec9889-8ddc-11e4-9efd-da6610fd24da:0 (5)
2014-12-28 02:24:25 9446 [Note] WSREP: Tables flushed.
WSREP_SST: [INFO] Preparing binlog files for transfer: (20141228 02:24:26.201)
mysql-bin.000015
2014-12-28 02:24:27 9446 [Note] WSREP: (2861d1d7, 'tcp://0.0.0.0:4567') turning message relay requesting off
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest1.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest2.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest3.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest4.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest5.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest6.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest7.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest8.ibd": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: send_files failed to open "/var/lib/mysql/mysql/innodb_index_stats.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/innodb_table_stats.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/slave_master_info.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/slave_relay_log_info.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/slave_worker_info.ibd": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
WSREP_SST: **[ERROR]** find/rsync returned code 123: (20141228 02:24:40.505)
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Failed to read from: **wsrep_sst_rsync** --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0'
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0': 255 (Unknown error 255)
2014-12-28 02:24:40 9446 [Note] WSREP: resuming provider at 5
2014-12-28 02:24:40 9446 [Note] WSREP: Provider resumed.
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0'
2014-12-28 02:24:40 9446 [Warning] WSREP: 0.0 (node3): State transfer to 1.0 (node2) failed: -255 (Unknown error 255)
2014-12-28 02:24:40 9446 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
2014-12-28 02:24:40 9446 [Note] WSREP: Member 0.0 (node3) synced with group.
2014-12-28 02:24:40 9446 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
2014-12-28 02:24:40 9446 [Note] WSREP: Synchronized with group, ready for connections
2014-12-28 02:24:40 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:41 9446 [Note] WSREP: forgetting 41f79045 (tcp://192.168.70.2:4567)
2014-12-28 02:24:41 9446 [Note] WSREP: Node 2861d1d7 state prim
2014-12-28 02:24:41 9446 [Note] WSREP: save pc into disk
2014-12-28 02:24:41 9446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2014-12-28 02:24:41 9446 [Note] WSREP: forgetting 41f79045 (tcp://192.168.70.2:4567)
2014-12-28 02:24:41 9446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2
2014-12-28 02:24:41 9446 [Note] WSREP: STATE EXCHANGE: sent state msg: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2
2014-12-28 02:24:41 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2 from 0 (node3)
2014-12-28 02:24:41 9446 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 2,
members = 1/1 (joined/total),
act_id = 0,
last_appl. = 0,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = 48ec9889-8ddc-11e4-9efd-da6610fd24da
2014-12-28 02:24:41 9446 [Note] WSREP: Flow-control interval: [16, 16]
2014-12-28 02:24:41 9446 [Note] WSREP: New cluster view: global state: 48ec9889-8ddc-11e4-9efd-da6610fd24da:0, view# 3: Primary, number of nodes: 1, my index: 0, protocol version 3
2014-12-28 02:24:41 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:41 9446 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-28 02:24:41 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:41 9446 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2014-12-28 02:24:41 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:46 9446 [Note] WSREP: cleaning up 41f79045 (tcp://192.168.70.2:4567)
Something is wrong with your permissions of the datadirectory as you can see from the following errors:
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest7.ibd": Permission denied (13)
Things I would check:
Are all data files in /var/lib/mysql owned by the mysql user?
do you have SELinux or AppArmor running? (It might be malconfigured).
Please note that a DONOR with rsync SST method will block reads/writes while it's the DONOR. This can cause downtime.
I suggest to look into Percona XtraBackup (The issues you have might be similar)
Related
Debian 10, Maridadb 10.3.26, Galera-3 25.3.31
I have a three node cluster. The nodes are named node3, node4, and node5. Node3 gets disconnected from the cluster on occasion.
If it picks node5 to recover from I get
2020-11-18 19:42:08 7 [Note] WSREP: Requesting state transfer: success, donor: 2
2020-11-18 19:42:08 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75720363
2020-11-18 19:42:08 17 [Note] WSREP: SST received: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989
2020-11-18 19:42:08 17 [Note] WSREP: wsrep_start_position set to '57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989'
2020-11-18 19:42:08 7 [Note] WSREP: Receiving IST: 23374 writesets, seqnos 75696989-75720363
2020-11-18 19:42:08 0 [Note] WSREP: 2.0 (node5): State transfer to 0.0 (node3) complete.
2020-11-18 19:42:08 0 [Note] WSREP: Member 2.0 (node5) synced with group.
2020-11-18 19:42:08 0 [Note] WSREP: (23249d11, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-11-18 19:42:15 0 [Warning] WSREP: Protocol violation. JOIN message sender 2.0 (node5) is not in state transfer (SYNCED). Message ignored.
after which node3 will sit forever never changing wsrep_ready to yes.
On the other hand, if node3 picks node4 I get all the same sort of messages except
[Warning] WSREP: Protocol violation. JOIN message sender 2.0 (node5) is not in state transfer (SYNCED). Message ignored.
does not appear and eventually node3 wsrep_ready becomes yes and the node starts to process queries.
Any idea how I much figure out the issue?
Here is some more data. This is an example of a successful join when it chooses node4 instead of node5:
2020-11-19 21:12:54 7 [Note] WSREP: State transfer required:
Group state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331
Local state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989
2020-11-19 21:12:54 7 [Note] WSREP: REPL Protocols: 9 (4, 2)
2020-11-19 21:12:54 7 [Note] WSREP: New cluster view: global state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331, view# 349: Primary, number of nodes: 3, my index: 2, protocol version 3
2020-11-19 21:12:54 7 [Warning] WSREP: Gap in state sequence. Need state transfer.
2020-11-19 21:12:56 7 [Note] WSREP: Prepared SST request: mysqldump|10.4.44.82:3360
2020-11-19 21:12:56 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-11-19 21:12:56 7 [Note] WSREP: Assign initial position for certification: 75815331, protocol version: 4
2020-11-19 21:12:56 0 [Note] WSREP: Service thread queue flushed.
2020-11-19 21:12:56 7 [Note] WSREP: IST receiver addr using tcp://x.y.z.a:4568
2020-11-19 21:12:56 7 [Note] WSREP: Prepared IST receiver, listening at: tcp://x.y.z.a:4568
2020-11-19 21:12:56 0 [Note] WSREP: Member 2.0 (node3) requested state transfer from '*any*'. Selected 0.0 (node4)(SYNCED) as donor.
2020-11-19 21:12:56 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 75815331)
2020-11-19 21:12:56 7 [Note] WSREP: Requesting state transfer: success, donor: 0
2020-11-19 21:12:56 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331
2020-11-19 21:12:56 0 [Note] WSREP: (fcbfdc45, 'tcp://0.0.0.0:4567') turning message relay requesting off
I have three node setup and running perfectly for the past months.
Recently I want to add another node in a different location but somehow I keep on getting errors.
At first, I was just following this tutorial (where I setup the first time few months ago) https://www.howtoforge.com/tutorial/how-to-install-and-configure-galera-cluster-on-ubuntu-1604/ I did not start all the nodes again from the beginning, I just has to find the file of /mysql/conf.d/galera.cnf in the other three nodes I added the new nodes ip into the previous three. So for the forth one I had the /etc/mysql/conf.d/galera.cnf setup like...
[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
# Galera Cluster Configuration
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://node1_ip,node2_ip,node3_ip,node4_ip"
# Galera Synchronization Configuration
wsrep_sst_method=rsync
# Galera Node Configuration
wsrep_node_address="xx.xx.xxx.xxx"
wsrep_node_name="Node4"
somehow I am getting this HUGE error,
Group state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
Local state: 00000000-0000-0000-0000-000000000000:-1
[Note] WSREP: New cluster view: global state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273, view# 122: Primary, number of nodes: 4, my
[Warning] WSREP: Gap in state sequence. Need state transfer.
[Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/' --parent '22828' ''
rsyncd version 3.1.1 starting, listening on port 4444
[Note] WSREP: Prepared SST request: rsync|xxx.node.4.ip:4444/rsync_sst
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
[Note] WSREP: REPL Protocols: 7 (3, 2)
[Note] WSREP: Assign initial position for certification: 36273, protocol version: 3
[Note] WSREP: Service thread queue flushed.
[Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
[Note] WSREP: Member 0.0 (Node4) requested state transfer from '*any*'. Selected 1.0 (Node1)(SYNCED) as donor.
[Note] WSREP: Shifting PRIMARY -> JOINER (TO: 36273)
[Note] WSREP: Requesting state transfer: success, donor: 1
[Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
[Note] WSREP: (7642cf37, 'tcp://0.0.0.0:4567') connection to peer 7642cf37 with addr tcp://xxx.node.4.ip:4567 timed out, no messages
[Note] WSREP: (7642cf37, 'tcp://0.0.0.0:4567') turning message relay requesting off
mariadb.service: Start operation timed out. Terminating.
Terminated
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 22875
sent 0 bytes received 0 bytes total size 0
WSREP_SST: [INFO] Joiner cleanup done.
[ERROR] WSREP: Process was aborted.
[ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/'
[ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
[ERROR] WSREP: SST failed: 2 (No such file or directory)
[ERROR] Aborting
Error in my_thread_global_end(): 1 threads didn't exit
mariadb.service: Main process exited, code=exited, status=1/FAILURE
Failed to start MariaDB 10.1.33 database server.
P.S for the older 3 nodes maria db version is 10.1.29 and the new node is 10.1.33
Thanks in advance for any suggestions.
I have a galera cluster (10.0.27) with 3 nodes, each on a dedicated server.
After the reboot of one of the servers, the node cannot join the cluster anymore neither perform a full SST.
Actually, it's like mysql 'misses' to launch some commands.
I have a second 'development' cluster with the same configuration working perfectly, I have no problem to add a node. I noticed a difference between the working cluster and not working when I add back a node for a full SST :
Node joining on working cluster :
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Quorum results:
11:44:52 mysqld: #011version = 4,
11:44:52 mysqld: #011component = PRIMARY,
11:44:52 mysqld: #011conf_id = 8,
11:44:52 mysqld: #011members = 2/3 (joined/total),
11:44:52 mysqld: #011act_id = 906976,
11:44:52 mysqld: #011last_appl. = -1,
11:44:52 mysqld: #011protocols = 0/7/3 (gcs/repl/appl),
11:44:52 mysqld: #011group UUID = 27ba4c4f-9b78-11e6-824c-f3b1e60fa202
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Flow-control interval: [28, 28]
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 906976)
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: State transfer required:
11:44:52 mysqld: #011Group state: 27ba4c4f-9b78-11e6-824c-f3b1e60fa202:906976
11:44:52 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: New cluster view: global state: 27ba4c4f-9b78-11e6-824c-f3b1e60fa202:906976, view# 9: Primary, number of nodes: 3, my index: 2, protocol version 3
11:44:52 mysqld: 170628 11:44:52 [Warning] WSREP: Gap in state sequence. Need state transfer.
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.***.***.**2' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '16472' --binlog '/var/log/mysql/mariadb-bin' '
**11:44:52 rsyncd[16514]: rsyncd version 3.1.1 starting, listening on port 4444**
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Prepared SST request: rsync|192.***.***.**2:4444/rsync_sst
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: REPL Protocols: 7 (3, 2)
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Assign initial position for certification: 906976, protocol version: 3
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Service thread queue flushed.
11:44:52 mysqld: 170628 11:44:52 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (27ba4c4f-9b78-11e6-824c-f3b1e60fa202): 1 (Operation not permitted)
11:44:52 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Member 2.0 (server-3) requested state transfer from '*any*'. Selected 0.0 (server1)(SYNCED) as donor.
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 906977)
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: Requesting state transfer: success, donor: 0
11:44:52 mysqld: 170628 11:44:52 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(27ba4c4f-9b78-11e6-824c-f3b1e60fa202:906976)
11:44:52 rsyncd[16531]: name lookup failed for 192.***.***.**1: Name or service not known
11:44:52 rsyncd[16531]: connect from UNKNOWN (192.***.***.**1)
11:44:52 rsyncd[16531]: rsync to rsync_sst/ from UNKNOWN (192.***.***.**1)
11:44:52 rsyncd[16531]: receiving file list
11:44:54 rsyncd[16553]: name lookup failed for 192.***.***.**1: Name or service not known
11:44:54 rsyncd[16553]: connect from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16531]: sent 114 bytes received 146847600 bytes total size 146810880
11:44:54 rsyncd[16553]: rsync to rsync_sst-log_dir/ from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16553]: receiving file list
11:44:54 rsyncd[16553]: sent 63 bytes received 100688095 bytes total size 100663296
11:44:54 rsyncd[16559]: name lookup failed for 192.***.***.**1: Name or service not known
11:44:54 rsyncd[16559]: connect from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16560]: name lookup failed for 192.***.***.**1: Name or service not known
11:44:54 rsyncd[16560]: connect from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16561]: name lookup failed for 192.***.***.**1: Name or service not known
11:44:54 rsyncd[16561]: connect from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16562]: name lookup failed for 192.***.***.**1: Name or service not known
11:44:54 rsyncd[16562]: connect from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16559]: rsync to rsync_sst/./db_1 from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16562]: rsync to rsync_sst/./db_2 from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16560]: rsync to rsync_sst/./db_3 from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16561]: rsync to rsync_sst/./db_3 from UNKNOWN (192.***.***.**1)
11:44:54 rsyncd[16560]: receiving file list
...
Node joining on non-working cluster :
13:36:28 mysqld: 170630 13:36:28 [Note] WSREP: Quorum results:
13:36:28 mysqld: #011version = 4,
13:36:28 mysqld: #011component = PRIMARY,
13:36:28 mysqld: #011conf_id = 514,
13:36:28 mysqld: #011members = 2/3 (joined/total),
13:36:28 mysqld: #011act_id = 242914778,
13:36:28 mysqld: #011last_appl. = -1,
13:36:28 mysqld: #011protocols = 0/7/3 (gcs/repl/appl),
13:36:28 mysqld: #011group UUID = 8119e584-9f83-11e6-b292-7a8102156c2d
13:36:28 mysqld: 170630 13:36:28 [Note] WSREP: Flow-control interval: [28, 28]
13:36:28 mysqld: 170630 13:36:28 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 242914778)
13:36:28 mysqld: 170630 13:36:28 [Note] WSREP: State transfer required:
13:36:28 mysqld: #011Group state: 8119e584-9f83-11e6-b292-7a8102156c2d:242914778
13:36:28 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
13:36:28 mysqld: 170630 13:36:28 [Note] WSREP: New cluster view: global state: 8119e584-9f83-1
1e6-b292-7a8102156c2d:242914778, view# 515: Primary, number of nodes: 3, my index: 2, protocol version 3
13:36:28 mysqld: 170630 13:36:28 [Warning] WSREP: Gap in state sequence. Need state transfer.
13:36:28 mysqld: 170630 13:36:28 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --add
ress '192.***.***.*11' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --pare
nt '13253' --binlog '/var/log/mysql/mariadb-bin' '
13:36:28 rsyncd[13316]: rsyncd version 3.1.1 starting, listening on port 4444
13:36:32 mysqld: 170630 13:36:32 [Note] WSREP: (85c5aae8, 'tcp://0.0.0.0:4567') turning messag
e relay requesting off
13:36:56 /etc/init.d/mysql[14935]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=
/etc/mysql/debian.cnf ping' resulted in
The difference is just after the line :
rsyncd[13316]: rsyncd version 3.1.1 starting, listening on port 4444
On the working cluster, the following line is
WSREP: Prepared SST request: rsync|192.***.***.**2:4444/rsync_sst
On the not working cluster this line doesn't appear, it's like the SST request is not made.
I can provide more information about the configuration if you think it can help to find the issue.
Thanks for your help !
Had the same issue, that's what I found:
The wsrep_sst_rsync is stuck in an endless loop. In my case, because the output of lsof -i :$rsync_port is empty. For some (unknown) reason, the lsof had the setgid bit set:
[dbserver1:~]# ls -l /usr/bin/lsof
-rwxr-sr-x 1 root root 163224 Oct 28 2015 /usr/bin/lsof
this caused an endless loop of wsrep_sst_rsync, as it checks if rsync could be started. Removing the flag causes the script to continue which eventually starts an SST.
The flag can be removed using:
[dbserver1:~]# chmod g-s /usr/bin/lsof
I have a node in the PXC that stopped, and now after a full sync from donor, it still does not start. The error log message from the donor is provided below:
When i ran the "show status like 'wsrep%'\G" on the donor node, it showed that the wsrep_local_state_comment changed from DONOR/DESYNCED to SYNCED. This i believe indicates that the full sync is complete. But i am not sure why the node won't start.
2015-10-17 16:26:46 13833 [Note] WSREP: STATE EXCHANGE: got state msg: cda64f2b-74a8-11e5-b965-36b9ac108f08 from 9 (db-kdb-5-2)
2015-10-17 16:26:46 13833 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 101,
members = 10/11 (joined/total),
act_id = 4281851091,
last_appl. = 4281850971,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 3248f155-08ac-11e4-a36a-1347fdb518d6
2015-10-17 16:26:46 13833 [Note] WSREP: Flow-control interval: [53, 53]
2015-10-17 16:26:46 13833 [Note] WSREP: New cluster view: global state: 3248f155-08ac-11e4-a36a-1347fdb518d6:4281851091, view# 102: Primary, number of nodes: 11, my index: 0, protocol version 3
2015-10-17 16:26:46 13833 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-10-17 16:26:46 13833 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-10-17 16:26:46 13833 [Note] WSREP: Service thread queue flushed.
2015-10-17 16:26:46 13833 [Note] WSREP: Assign initial position for certification: 4281851091, protocol version: 3
2015-10-17 16:26:46 13833 [Note] WSREP: Service thread queue flushed.
2015-10-17 16:26:46 13833 [Note] WSREP: Member 9.0 (db-node-1) requested state transfer from ‘db-node1'. Selected 0.0 (db-node-1)(SYNCED) as donor.
2015-10-17 16:26:46 13833 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 4281851185)
2015-10-17 16:26:46 13833 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-10-17 16:26:46 13833 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address ‘192.168.2.51:4444/xtrabackup_sst//1' --auth ‘user:pass' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '3248f155-08ac-11e4-a36a-1347fdb518d6:4281851185''
2015-10-17 16:26:47 13833 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] Streaming with xbstream (20151017 16:26:47.542)
WSREP_SST: [INFO] Using socat as streamer (20151017 16:26:47.546)
WSREP_SST: [INFO] Using /tmp/tmp.ZQBX8Kolho as innobackupex temporary directory (20151017 16:26:47.560)
WSREP_SST: [INFO] Streaming GTID file before SST (20151017 16:26:47.567)
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:10.1.5.51:4444; RC=( ${PIPESTATUS[#]} ) (20151017 16:26:47.570)
WSREP_SST: [INFO] Sleeping before data transfer for SST (20151017 16:26:47.577)
2015-10-17 16:26:47 13833 [Note] WSREP: (1e75ae0c, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.2.51 4444 (20151017 16:26:57.582)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf --defaults-group=mysqld --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:192.168.2.51:4444; RC=( ${PIPESTATUS[#]} ) (20151017 16:26:57.586)
2015/10/17 16:26:59 socat[4849] E write(3, 0x17741d0, 8192): Broken pipe
WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (20151017 16:26:59.875)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20151017 16:26:59.879)
WSREP_SST: [INFO] Cleaning up temporary directories (20151017 16:26:59.884)
2015-10-17 16:26:59 13833 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address ‘192.168.2.51:4444/xtrabackup_sst//1' --auth ‘user:pass' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '3248f155-08ac-11e4-a36a-1347fdb518d6:4281851185'
2015-10-17 16:26:59 13833 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address ‘192.168.2.51:4444/xtrabackup_sst//1' --auth ‘user:pass' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '3248f155-08ac-11e4-a36a-1347fdb518d6:4281851185': 22 (Invalid argument)
2015-10-17 16:26:59 13833 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address ’192.168.2.51:4444/xtrabackup_sst//1' --auth ‘user:pass' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '3248f155-08ac-11e4-a36a-1347fdb518d6:4281851185'
2015-10-17 16:26:59 13833 [Warning] WSREP: 0.0 (db-node-2): State transfer to 9.0 (db-node-1) failed: -22 (Invalid argument)
2015-10-17 16:26:59 13833 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 4281853007)
2015-10-17 16:26:59 13833 [Note] WSREP: (1e75ae0c, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.4:4567' pointing to uuid 1e75ae0c is blacklisted, skipping
2015-10-17 16:26:59 13833 [Note] WSREP: (1e75ae0c, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.4.:4567' pointing to uuid 1e75ae0c is blacklisted, skipping
2015-10-17 16:26:59 13833 [Note] WSREP: (1e75ae0c, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.4:4567' pointing to uuid 1e75ae0c is blacklisted, skipping
2015-10-17 16:26:59 13833 [Note] WSREP: (1e75ae0c, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.4:4567' pointing to uuid 1e75ae0c is blacklisted, skipping
Ok so I have a second node I am trying to add to a working galera mysql server as another node...configs here
Node A(working)
[server]
[mysqld]
[embedded]
[mysqld-5.5]
[mariadb]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=172.16.1.20
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="controller_cluster"
wsrep_cluster_address="gcomm://"
wsrep_sst_receive_addres="172.16.1.20"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
wsrep_sst_auth=wsrep_sst:password
[mariadb-5.5]
Node B(wont start)
[server]
[mysqld]
skip-name-resolve
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
[embedded]
[mysqld-5.5]
[mariadb]
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=172.16.1.21
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="controller_cluster"
wsrep_cluster_address="gcomm://172.16.1.20"
wsrep_sst_receive_addres="172.16.1.21"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
wsrep_sst_auth=wsrep_sst:password
[mariadb-5.5]
Node A Permissions on /var/lib/mysql
-rw-rw----. 1 mysql mysql 16384 Mar 4 23:54 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Mar 4 23:54 aria_log_control
-rw-r-----. 1 mysql root 283162 Mar 5 17:49 db01.deg.pod1.err
-rw-rw----. 1 mysql mysql 5 Mar 4 23:54 db01.deg.pod1.pid
-rw-------. 1 mysql mysql 134219040 Mar 5 17:48 galera.cache
-rw-rw----. 1 mysql mysql 104 Mar 5 17:48 grastate.dat
-rw-rw----. 1 mysql mysql 12582912 Mar 4 23:54 ibdata1
-rw-rw----. 1 mysql mysql 5242880 Mar 4 23:54 ib_logfile0
-rw-rw----. 1 mysql mysql 5242880 Mar 4 22:30 ib_logfile1
drwx------. 2 mysql mysql 4096 Mar 4 22:59 mysql
srwxrwxrwx. 1 mysql mysql 0 Mar 4 23:54 mysql.sock
drwx------. 2 root root 4096 Mar 4 22:59 performance_schema
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:11 RPM_UPGRADE_HISTORY
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:11 RPM_UPGRADE_MARKER-LAST
drwxr-xr-x. 2 mysql mysql 4096 Mar 4 22:11 test
drwx------. 2 mysql mysql 4096 Mar 5 17:35 tt
Node B Permissions on /var/lib/mysql
-rw-rw----. 1 mysql mysql 16384 Mar 5 17:49 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Mar 5 17:49 aria_log_control
-rw-r-----. 1 mysql root 0 Mar 5 17:49 db02.deg.pod1.err
-rw-------. 1 mysql mysql 134219040 Mar 5 17:49 galera.cache
-rw-rw----. 1 mysql mysql 104 Mar 5 17:49 grastate.dat
-rw-rw----. 1 mysql mysql 12582912 Mar 5 17:49 ibdata1
-rw-rw----. 1 mysql mysql 5242880 Mar 5 17:49 ib_logfile0
-rw-rw----. 1 mysql mysql 5242880 Mar 5 17:49 ib_logfile1
drwx------. 2 mysql mysql 4096 Mar 4 23:10 mysql
srwxrwxrwx 1 mysql mysql 0 Mar 5 17:49 mysql.sock
-rw-------. 1 root root 107 Mar 4 23:10 nohup.out
-rw-r--r-- 1 root root 269455 Mar 5 03:42 out.log
drwx------ 2 root root 4096 Mar 5 03:20 performance_schema
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:14 RPM_UPGRADE_HISTORY
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:14 RPM_UPGRADE_MARKER-LAST
drwxr-xr-x. 2 mysql mysql 4096 Mar 4 22:14 test
drwx------ 2 mysql mysql 4096 Mar 5 17:36 tt
-rw------- 1 root root 0 Mar 5 03:52 wsrep_recovery.hh4i9
Password is also the same on both ends for mysql user and for root user.
** Failure log on Node B **
140305 17:49:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140305 17:49:39 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.A8nOzH' --pid-file='/var/lib/mysql/db02.deg.pod1-recover.pid'
nohup: ignoring input
140305 17:49:39 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:39 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 mysqld_safe WSREP: Recovered position bce8f04b-a41a-11e3-b010-4ba4a408598c:0
140305 17:49:41 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 [Note] WSREP: wsrep_start_position var submitted: 'bce8f04b-a41a-11e3-b010-4ba4a408598c:0'
140305 17:49:41 [Note] WSREP: Read nil XID from storage engines, skipping position init
140305 17:49:41 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
140305 17:49:41 [Note] WSREP: wsrep_load(): Galera 25.3.2(r170) by Codership Oy <info#codership.com> loaded successfully.
140305 17:49:41 [Note] WSREP: CRC-32C: using hardware acceleration.
140305 17:49:41 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
140305 17:49:41 [Note] WSREP: Passing config to GCS: base_host = 172.16.1.21; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2147483647; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5
140305 17:49:41 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
140305 17:49:41 [Note] WSREP: wsrep_sst_grab()
140305 17:49:41 [Note] WSREP: Start replication
140305 17:49:41 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
140305 17:49:41 [Note] WSREP: protonet asio version 0
140305 17:49:41 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
140305 17:49:41 [Note] WSREP: backend: asio
140305 17:49:41 [Note] WSREP: GMCast version 0
140305 17:49:41 [Note] WSREP: (7036f7c8-a4b8-11e3-97c3-866382997e69, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
140305 17:49:41 [Note] WSREP: (7036f7c8-a4b8-11e3-97c3-866382997e69, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
140305 17:49:41 [Note] WSREP: EVS version 0
140305 17:49:41 [Note] WSREP: PC version 0
140305 17:49:41 [Note] WSREP: gcomm: connecting to group 'controller_cluster', peer '172.16.1.20:'
140305 17:49:41 [Note] WSREP: declaring 3f183cba-a422-11e3-b1c7-52f230abd39f stable
140305 17:49:41 [Note] WSREP: Node 3f183cba-a422-11e3-b1c7-52f230abd39f state prim
140305 17:49:41 [Note] WSREP: view(view_id(PRIM,3f183cba-a422-11e3-b1c7-52f230abd39f,48) memb {
3f183cba-a422-11e3-b1c7-52f230abd39f,0
7036f7c8-a4b8-11e3-97c3-866382997e69,0
} joined {
} left {
} partitioned {
})
140305 17:49:42 [Note] WSREP: gcomm: connected
140305 17:49:42 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
140305 17:49:42 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
140305 17:49:42 [Note] WSREP: Opened channel 'controller_cluster'
140305 17:49:42 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
140305 17:49:42 [Note] WSREP: Waiting for SST to complete.
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: sent state msg: 52478b99-a4b8-11e3-9283-8207651d087e
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: got state msg: 52478b99-a4b8-11e3-9283-8207651d087e from 0 (db01.deg.pod1)
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: got state msg: 52478b99-a4b8-11e3-9283-8207651d087e from 1 (db02.deg.pod1)
140305 17:49:42 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 47,
members = 1/2 (joined/total),
act_id = 1,
last_appl. = -1,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = bce8f04b-a41a-11e3-b010-4ba4a408598c
140305 17:49:42 [Note] WSREP: Flow-control interval: [23, 23]
140305 17:49:42 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
140305 17:49:42 [Note] WSREP: State transfer required:
Group state: bce8f04b-a41a-11e3-b010-4ba4a408598c:1
Local state: 00000000-0000-0000-0000-000000000000:-1
140305 17:49:42 [Note] WSREP: New cluster view: global state: bce8f04b-a41a-11e3-b010-4ba4a408598c:1, view# 48: Primary, number of nodes: 2, my index: 1, protocol version 2
140305 17:49:42 [Warning] WSREP: Gap in state sequence. Need state transfer.
140305 17:49:44 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.16.1.21' --auth 'wsrep_sst:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '13639''
140305 17:49:44 [Note] WSREP: Prepared SST request: rsync|172.16.1.21:4444/rsync_sst
140305 17:49:44 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140305 17:49:44 [Note] WSREP: REPL Protocols: 5 (3, 1)
140305 17:49:44 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
140305 17:49:44 [Note] WSREP: Service thread queue flushed.
140305 17:49:44 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bce8f04b-a41a-11e3-b010-4ba4a408598c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():445. IST will be unavailable.
140305 17:49:44 [Note] WSREP: Node 1.0 (db02.deg.pod1) requested state transfer from '*any*'. Selected 0.0 (db01.deg.pod1)(SYNCED) as donor.
140305 17:49:44 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
140305 17:49:44 [Note] WSREP: Requesting state transfer: success, donor: 0
140305 17:49:45 [Warning] WSREP: 0.0 (db01.deg.pod1): State transfer to 1.0 (db02.deg.pod1) failed: -1 (Operation not permitted)
140305 17:49:45 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
140305 17:49:45 [Note] WSREP: gcomm: terminating thread
140305 17:49:45 [Note] WSREP: gcomm: joining thread
140305 17:49:45 [Note] WSREP: gcomm: closing backend
140305 17:49:46 [Note] WSREP: view(view_id(NON_PRIM,3f183cba-a422-11e3-b1c7-52f230abd39f,48) memb {
7036f7c8-a4b8-11e3-97c3-866382997e69,0
} joined {
} left {
} partitioned {
3f183cba-a422-11e3-b1c7-52f230abd39f,0
})
140305 17:49:46 [Note] WSREP: view((empty))
140305 17:49:46 [Note] WSREP: gcomm: closed
140305 17:49:46 [Note] WSREP: /usr/sbin/mysqld: Terminated.
140305 17:49:46 mysqld_safe mysqld from pid file /var/lib/mysql/db02.deg.pod1.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:13639) terminated unexpectedly. (20140305 17:49:47.601)
WSREP_SST: [INFO] Joiner cleanup. (20140305 17:49:47.603)
WSREP_SST: [INFO] Joiner cleanup done. (20140305 17:49:48.110)
One possible reason for nodes failing to add is that the existing nodes don't expect them. In your configuration files, you may just need to point each node to the rest.
For example, try using wsrep_cluster_address="gcomm://"172.16.1.20,172.16.1.21" in both galera.cnf files.