I have a two separate Galera clusters, will call them C1 and C2.
C1 is a main cluster and has a three nodes, C2 is emergency and has only one node. C1 connected with C2 by GTID replication for providing completely identity.
The main problem that I can't increase a number of nodes in C2 cluster. I get these errors:
2016-08-02 12:54:00 19067 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 2th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 3th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 4th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [ERROR] WSREP: Failed to apply trx: source: 377c1c2e-5885-11e6-aed8-1e04b1e53bff version: 3 local: 0 state: APPLYING flags: 1 conn_id: 346 trx
_id: 23137618 seqnos (l: 5, g: 12305593, s: 12305592, d: 12305592, ts: 16848639299414398)
2016-08-02 13:00:58 19067 [ERROR] WSREP: Failed to apply trx 12305593 4 times
2016-08-02 13:00:58 19067 [ERROR] WSREP: Node consistency compromized, aborting...
2016-08-02 13:00:58 19067 [Note] WSREP: Closing send monitor...
2016-08-02 13:00:58 19067 [Note] WSREP: Closed send monitor.
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: terminating thread
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: joining thread
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: closing backend
The main settings that relates to the issue:
C1 and C2:
binlog_format=ROW
wsrep_sst_method=xtrabackup-v2
server_id = 1 (C1); server_id = 101 (C2)
wsrep_OSU_method = 'TOI'
What is most interesting, that with simple async replication (not GTID) all worked properly. Have you any ideas, guys?
Related
Debian 10, Maridadb 10.3.26, Galera-3 25.3.31
I have a three node cluster. The nodes are named node3, node4, and node5. Node3 gets disconnected from the cluster on occasion.
If it picks node5 to recover from I get
2020-11-18 19:42:08 7 [Note] WSREP: Requesting state transfer: success, donor: 2
2020-11-18 19:42:08 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75720363
2020-11-18 19:42:08 17 [Note] WSREP: SST received: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989
2020-11-18 19:42:08 17 [Note] WSREP: wsrep_start_position set to '57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989'
2020-11-18 19:42:08 7 [Note] WSREP: Receiving IST: 23374 writesets, seqnos 75696989-75720363
2020-11-18 19:42:08 0 [Note] WSREP: 2.0 (node5): State transfer to 0.0 (node3) complete.
2020-11-18 19:42:08 0 [Note] WSREP: Member 2.0 (node5) synced with group.
2020-11-18 19:42:08 0 [Note] WSREP: (23249d11, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-11-18 19:42:15 0 [Warning] WSREP: Protocol violation. JOIN message sender 2.0 (node5) is not in state transfer (SYNCED). Message ignored.
after which node3 will sit forever never changing wsrep_ready to yes.
On the other hand, if node3 picks node4 I get all the same sort of messages except
[Warning] WSREP: Protocol violation. JOIN message sender 2.0 (node5) is not in state transfer (SYNCED). Message ignored.
does not appear and eventually node3 wsrep_ready becomes yes and the node starts to process queries.
Any idea how I much figure out the issue?
Here is some more data. This is an example of a successful join when it chooses node4 instead of node5:
2020-11-19 21:12:54 7 [Note] WSREP: State transfer required:
Group state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331
Local state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989
2020-11-19 21:12:54 7 [Note] WSREP: REPL Protocols: 9 (4, 2)
2020-11-19 21:12:54 7 [Note] WSREP: New cluster view: global state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331, view# 349: Primary, number of nodes: 3, my index: 2, protocol version 3
2020-11-19 21:12:54 7 [Warning] WSREP: Gap in state sequence. Need state transfer.
2020-11-19 21:12:56 7 [Note] WSREP: Prepared SST request: mysqldump|10.4.44.82:3360
2020-11-19 21:12:56 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-11-19 21:12:56 7 [Note] WSREP: Assign initial position for certification: 75815331, protocol version: 4
2020-11-19 21:12:56 0 [Note] WSREP: Service thread queue flushed.
2020-11-19 21:12:56 7 [Note] WSREP: IST receiver addr using tcp://x.y.z.a:4568
2020-11-19 21:12:56 7 [Note] WSREP: Prepared IST receiver, listening at: tcp://x.y.z.a:4568
2020-11-19 21:12:56 0 [Note] WSREP: Member 2.0 (node3) requested state transfer from '*any*'. Selected 0.0 (node4)(SYNCED) as donor.
2020-11-19 21:12:56 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 75815331)
2020-11-19 21:12:56 7 [Note] WSREP: Requesting state transfer: success, donor: 0
2020-11-19 21:12:56 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331
2020-11-19 21:12:56 0 [Note] WSREP: (fcbfdc45, 'tcp://0.0.0.0:4567') turning message relay requesting off
I have three node setup and running perfectly for the past months.
Recently I want to add another node in a different location but somehow I keep on getting errors.
At first, I was just following this tutorial (where I setup the first time few months ago) https://www.howtoforge.com/tutorial/how-to-install-and-configure-galera-cluster-on-ubuntu-1604/ I did not start all the nodes again from the beginning, I just has to find the file of /mysql/conf.d/galera.cnf in the other three nodes I added the new nodes ip into the previous three. So for the forth one I had the /etc/mysql/conf.d/galera.cnf setup like...
[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
# Galera Cluster Configuration
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://node1_ip,node2_ip,node3_ip,node4_ip"
# Galera Synchronization Configuration
wsrep_sst_method=rsync
# Galera Node Configuration
wsrep_node_address="xx.xx.xxx.xxx"
wsrep_node_name="Node4"
somehow I am getting this HUGE error,
Group state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
Local state: 00000000-0000-0000-0000-000000000000:-1
[Note] WSREP: New cluster view: global state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273, view# 122: Primary, number of nodes: 4, my
[Warning] WSREP: Gap in state sequence. Need state transfer.
[Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/' --parent '22828' ''
rsyncd version 3.1.1 starting, listening on port 4444
[Note] WSREP: Prepared SST request: rsync|xxx.node.4.ip:4444/rsync_sst
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
[Note] WSREP: REPL Protocols: 7 (3, 2)
[Note] WSREP: Assign initial position for certification: 36273, protocol version: 3
[Note] WSREP: Service thread queue flushed.
[Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
[Note] WSREP: Member 0.0 (Node4) requested state transfer from '*any*'. Selected 1.0 (Node1)(SYNCED) as donor.
[Note] WSREP: Shifting PRIMARY -> JOINER (TO: 36273)
[Note] WSREP: Requesting state transfer: success, donor: 1
[Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
[Note] WSREP: (7642cf37, 'tcp://0.0.0.0:4567') connection to peer 7642cf37 with addr tcp://xxx.node.4.ip:4567 timed out, no messages
[Note] WSREP: (7642cf37, 'tcp://0.0.0.0:4567') turning message relay requesting off
mariadb.service: Start operation timed out. Terminating.
Terminated
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 22875
sent 0 bytes received 0 bytes total size 0
WSREP_SST: [INFO] Joiner cleanup done.
[ERROR] WSREP: Process was aborted.
[ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/'
[ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
[ERROR] WSREP: SST failed: 2 (No such file or directory)
[ERROR] Aborting
Error in my_thread_global_end(): 1 threads didn't exit
mariadb.service: Main process exited, code=exited, status=1/FAILURE
Failed to start MariaDB 10.1.33 database server.
P.S for the older 3 nodes maria db version is 10.1.29 and the new node is 10.1.33
Thanks in advance for any suggestions.
I've setup 3 node galera cluster for my application, there is a certain stored procedure in my application that creates a table temporarily(Not TEMPORARY TABLE), the table is created dynamically and executed. Some CRUD operation is done on the table muptiple times and after its usage is completed it is dropped at the end of the stored procedure.
The problem is that everytime I run this particular stored procedure 2 out of my 3 node fails and shuts down automatically.
Here is the Log from the node that fails :-
05 seqnos (l: 38, g: 47, s: 46, d: 46, ts: 186965868554819)
150423 8:31:35 [ERROR] WSREP: Failed to apply trx 47 4 times
150423 8:31:35 [ERROR] WSREP: Node consistency compromized, aborting...
150423 8:31:35 [Note] WSREP: Closing send monitor...
150423 8:31:35 [Note] WSREP: Closed send monitor.
150423 8:31:35 [Note] WSREP: gcomm: terminating thread
150423 8:31:35 [Note] WSREP: gcomm: joining thread
150423 8:31:35 [Note] WSREP: gcomm: closing backend
150423 8:31:35 [Note] WSREP: view(view_id(NON_PRIM,4d2adf77-e972-11e4-be7e-6bcb3d5c882f,3) memb {
849bbb26-e981-11e4-9f9f-f37ebca03acf,0
} joined {
} left {
} partitioned {
4d2adf77-e972-11e4-be7e-6bcb3d5c882f,0
634f32cb-e972-11e4-a226-bf661daf79d7,0
})
150423 8:31:35 [Note] WSREP: view((empty))
150423 8:31:35 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
150423 8:31:35 [Note] WSREP: gcomm: closed
150423 8:31:35 [Note] WSREP: Flow-control interval: [16, 16]
150423 8:31:35 [Note] WSREP: Received NON-PRIMARY.
150423 8:31:35 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 47)
150423 8:31:35 [Note] WSREP: Received self-leave message.
150423 8:31:35 [Note] WSREP: Flow-control interval: [0, 0]
150423 8:31:35 [Note] WSREP: Received SELF-LEAVE. Closing connection.
150423 8:31:35 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 47)
150423 8:31:35 [Note] WSREP: RECV thread exiting 0: Success
150423 8:31:35 [Note] WSREP: recv_thread() joined.
150423 8:31:35 [Note] WSREP: Closing replication queue.
150423 8:31:35 [Note] WSREP: Closing slave action queue.
150423 8:31:35 [Note] WSREP: /usr/sbin/mysqld: Terminated.
150423 08:31:35 mysqld_safe Number of processes running now: 0
150423 08:31:35 mysqld_safe WSREP: not restarting wsrep node automatically
150423 08:31:35 mysqld_safe mysqld from pid file /mnt/mysql/db3-prod.pid ended
is There any particular I can solve this without modifying the internal structure of the stored procedure.
The output of
SHOW VARIABLES LIKE 'default%'; is
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| default_master_connection | |
| default_regex_flags | |
| default_storage_engine | InnoDB |
| default_week_format | 0 |
+---------------------------+--------+
I'm sure there is a simple fix for this, but forgive me I'm new at PXC. I'm using rsync to transfer state of the bootstrapping node to node2. node2 is the node I am joining to the cluster. I originally tried Xtrabackup, but ran into problems which I will explore at another time. For now I'm using rsync for what I thought would be simplicity. If you scroll down to the [ERROR] you will see where the problems are causing the State Transfer to be interrupted. What could be causing this?
*WSREP_SST: **[ERROR]** find/rsync returned code 123: (20141228 02:24:40.505)
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Failed to read from: **wsrep_sst_rsync** --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket*
2014-12-28 02:24:24 9446 [Note] WSREP: (2861d1d7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2014-12-28 02:24:24 9446 [Note] WSREP: declaring 41f79045 at tcp://192.168.70.2:4567 stable
2014-12-28 02:24:24 9446 [Note] WSREP: Node 2861d1d7 state prim
2014-12-28 02:24:24 9446 [Note] WSREP: save pc into disk
2014-12-28 02:24:24 9446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2014-12-28 02:24:24 9446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 42495aba-8e30-11e4-9596-c702faf22ad0
2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: sent state msg: 42495aba-8e30-11e4-9596-c702faf22ad0
2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 42495aba-8e30-11e4-9596-c702faf22ad0 from 0 (node3)
2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 42495aba-8e30-11e4-9596-c702faf22ad0 from 1 (node2)
2014-12-28 02:24:24 9446 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 1,
members = 1/2 (joined/total),
act_id = 0,
last_appl. = 0,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = 48ec9889-8ddc-11e4-9efd-da6610fd24da
2014-12-28 02:24:24 9446 [Note] WSREP: Flow-control interval: [23, 23]
2014-12-28 02:24:24 9446 [Note] WSREP: New cluster view: global state: 48ec9889-8ddc-11e4-9efd-da6610fd24da:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
2014-12-28 02:24:24 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:24 9446 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-28 02:24:24 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:24 9446 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2014-12-28 02:24:24 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:25 9446 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node3)(SYNCED) as donor.
2014-12-28 02:24:25 9446 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
2014-12-28 02:24:25 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:25 9446 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0''
2014-12-28 02:24:25 9446 [Note] WSREP: sst_donor_thread signaled with 0
2014-12-28 02:24:25 9446 [Note] WSREP: Flushing tables for SST...
2014-12-28 02:24:25 9446 [Note] WSREP: Provider paused at 48ec9889-8ddc-11e4-9efd-da6610fd24da:0 (5)
2014-12-28 02:24:25 9446 [Note] WSREP: Tables flushed.
WSREP_SST: [INFO] Preparing binlog files for transfer: (20141228 02:24:26.201)
mysql-bin.000015
2014-12-28 02:24:27 9446 [Note] WSREP: (2861d1d7, 'tcp://0.0.0.0:4567') turning message relay requesting off
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest1.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest2.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest3.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest4.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest5.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest6.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest7.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest8.ibd": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: send_files failed to open "/var/lib/mysql/mysql/innodb_index_stats.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/innodb_table_stats.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/slave_master_info.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/slave_relay_log_info.ibd": Permission denied (13)
rsync: send_files failed to open "/var/lib/mysql/mysql/slave_worker_info.ibd": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
WSREP_SST: **[ERROR]** find/rsync returned code 123: (20141228 02:24:40.505)
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Failed to read from: **wsrep_sst_rsync** --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0'
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0': 255 (Unknown error 255)
2014-12-28 02:24:40 9446 [Note] WSREP: resuming provider at 5
2014-12-28 02:24:40 9446 [Note] WSREP: Provider resumed.
2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0'
2014-12-28 02:24:40 9446 [Warning] WSREP: 0.0 (node3): State transfer to 1.0 (node2) failed: -255 (Unknown error 255)
2014-12-28 02:24:40 9446 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 0)
2014-12-28 02:24:40 9446 [Note] WSREP: Member 0.0 (node3) synced with group.
2014-12-28 02:24:40 9446 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
2014-12-28 02:24:40 9446 [Note] WSREP: Synchronized with group, ready for connections
2014-12-28 02:24:40 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:41 9446 [Note] WSREP: forgetting 41f79045 (tcp://192.168.70.2:4567)
2014-12-28 02:24:41 9446 [Note] WSREP: Node 2861d1d7 state prim
2014-12-28 02:24:41 9446 [Note] WSREP: save pc into disk
2014-12-28 02:24:41 9446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2014-12-28 02:24:41 9446 [Note] WSREP: forgetting 41f79045 (tcp://192.168.70.2:4567)
2014-12-28 02:24:41 9446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2
2014-12-28 02:24:41 9446 [Note] WSREP: STATE EXCHANGE: sent state msg: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2
2014-12-28 02:24:41 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2 from 0 (node3)
2014-12-28 02:24:41 9446 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 2,
members = 1/1 (joined/total),
act_id = 0,
last_appl. = 0,
protocols = 0/6/3 (gcs/repl/appl),
group UUID = 48ec9889-8ddc-11e4-9efd-da6610fd24da
2014-12-28 02:24:41 9446 [Note] WSREP: Flow-control interval: [16, 16]
2014-12-28 02:24:41 9446 [Note] WSREP: New cluster view: global state: 48ec9889-8ddc-11e4-9efd-da6610fd24da:0, view# 3: Primary, number of nodes: 1, my index: 0, protocol version 3
2014-12-28 02:24:41 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-28 02:24:41 9446 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-28 02:24:41 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:41 9446 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2014-12-28 02:24:41 9446 [Note] WSREP: Service thread queue flushed.
2014-12-28 02:24:46 9446 [Note] WSREP: cleaning up 41f79045 (tcp://192.168.70.2:4567)
Something is wrong with your permissions of the datadirectory as you can see from the following errors:
rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest7.ibd": Permission denied (13)
Things I would check:
Are all data files in /var/lib/mysql owned by the mysql user?
do you have SELinux or AppArmor running? (It might be malconfigured).
Please note that a DONOR with rsync SST method will block reads/writes while it's the DONOR. This can cause downtime.
I suggest to look into Percona XtraBackup (The issues you have might be similar)
Ok so I have a second node I am trying to add to a working galera mysql server as another node...configs here
Node A(working)
[server]
[mysqld]
[embedded]
[mysqld-5.5]
[mariadb]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=172.16.1.20
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="controller_cluster"
wsrep_cluster_address="gcomm://"
wsrep_sst_receive_addres="172.16.1.20"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
wsrep_sst_auth=wsrep_sst:password
[mariadb-5.5]
Node B(wont start)
[server]
[mysqld]
skip-name-resolve
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
[embedded]
[mysqld-5.5]
[mariadb]
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=172.16.1.21
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="controller_cluster"
wsrep_cluster_address="gcomm://172.16.1.20"
wsrep_sst_receive_addres="172.16.1.21"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
wsrep_sst_auth=wsrep_sst:password
[mariadb-5.5]
Node A Permissions on /var/lib/mysql
-rw-rw----. 1 mysql mysql 16384 Mar 4 23:54 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Mar 4 23:54 aria_log_control
-rw-r-----. 1 mysql root 283162 Mar 5 17:49 db01.deg.pod1.err
-rw-rw----. 1 mysql mysql 5 Mar 4 23:54 db01.deg.pod1.pid
-rw-------. 1 mysql mysql 134219040 Mar 5 17:48 galera.cache
-rw-rw----. 1 mysql mysql 104 Mar 5 17:48 grastate.dat
-rw-rw----. 1 mysql mysql 12582912 Mar 4 23:54 ibdata1
-rw-rw----. 1 mysql mysql 5242880 Mar 4 23:54 ib_logfile0
-rw-rw----. 1 mysql mysql 5242880 Mar 4 22:30 ib_logfile1
drwx------. 2 mysql mysql 4096 Mar 4 22:59 mysql
srwxrwxrwx. 1 mysql mysql 0 Mar 4 23:54 mysql.sock
drwx------. 2 root root 4096 Mar 4 22:59 performance_schema
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:11 RPM_UPGRADE_HISTORY
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:11 RPM_UPGRADE_MARKER-LAST
drwxr-xr-x. 2 mysql mysql 4096 Mar 4 22:11 test
drwx------. 2 mysql mysql 4096 Mar 5 17:35 tt
Node B Permissions on /var/lib/mysql
-rw-rw----. 1 mysql mysql 16384 Mar 5 17:49 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Mar 5 17:49 aria_log_control
-rw-r-----. 1 mysql root 0 Mar 5 17:49 db02.deg.pod1.err
-rw-------. 1 mysql mysql 134219040 Mar 5 17:49 galera.cache
-rw-rw----. 1 mysql mysql 104 Mar 5 17:49 grastate.dat
-rw-rw----. 1 mysql mysql 12582912 Mar 5 17:49 ibdata1
-rw-rw----. 1 mysql mysql 5242880 Mar 5 17:49 ib_logfile0
-rw-rw----. 1 mysql mysql 5242880 Mar 5 17:49 ib_logfile1
drwx------. 2 mysql mysql 4096 Mar 4 23:10 mysql
srwxrwxrwx 1 mysql mysql 0 Mar 5 17:49 mysql.sock
-rw-------. 1 root root 107 Mar 4 23:10 nohup.out
-rw-r--r-- 1 root root 269455 Mar 5 03:42 out.log
drwx------ 2 root root 4096 Mar 5 03:20 performance_schema
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:14 RPM_UPGRADE_HISTORY
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:14 RPM_UPGRADE_MARKER-LAST
drwxr-xr-x. 2 mysql mysql 4096 Mar 4 22:14 test
drwx------ 2 mysql mysql 4096 Mar 5 17:36 tt
-rw------- 1 root root 0 Mar 5 03:52 wsrep_recovery.hh4i9
Password is also the same on both ends for mysql user and for root user.
** Failure log on Node B **
140305 17:49:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140305 17:49:39 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.A8nOzH' --pid-file='/var/lib/mysql/db02.deg.pod1-recover.pid'
nohup: ignoring input
140305 17:49:39 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:39 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 mysqld_safe WSREP: Recovered position bce8f04b-a41a-11e3-b010-4ba4a408598c:0
140305 17:49:41 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 [Note] WSREP: wsrep_start_position var submitted: 'bce8f04b-a41a-11e3-b010-4ba4a408598c:0'
140305 17:49:41 [Note] WSREP: Read nil XID from storage engines, skipping position init
140305 17:49:41 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
140305 17:49:41 [Note] WSREP: wsrep_load(): Galera 25.3.2(r170) by Codership Oy <info#codership.com> loaded successfully.
140305 17:49:41 [Note] WSREP: CRC-32C: using hardware acceleration.
140305 17:49:41 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
140305 17:49:41 [Note] WSREP: Passing config to GCS: base_host = 172.16.1.21; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2147483647; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5
140305 17:49:41 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
140305 17:49:41 [Note] WSREP: wsrep_sst_grab()
140305 17:49:41 [Note] WSREP: Start replication
140305 17:49:41 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
140305 17:49:41 [Note] WSREP: protonet asio version 0
140305 17:49:41 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
140305 17:49:41 [Note] WSREP: backend: asio
140305 17:49:41 [Note] WSREP: GMCast version 0
140305 17:49:41 [Note] WSREP: (7036f7c8-a4b8-11e3-97c3-866382997e69, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
140305 17:49:41 [Note] WSREP: (7036f7c8-a4b8-11e3-97c3-866382997e69, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
140305 17:49:41 [Note] WSREP: EVS version 0
140305 17:49:41 [Note] WSREP: PC version 0
140305 17:49:41 [Note] WSREP: gcomm: connecting to group 'controller_cluster', peer '172.16.1.20:'
140305 17:49:41 [Note] WSREP: declaring 3f183cba-a422-11e3-b1c7-52f230abd39f stable
140305 17:49:41 [Note] WSREP: Node 3f183cba-a422-11e3-b1c7-52f230abd39f state prim
140305 17:49:41 [Note] WSREP: view(view_id(PRIM,3f183cba-a422-11e3-b1c7-52f230abd39f,48) memb {
3f183cba-a422-11e3-b1c7-52f230abd39f,0
7036f7c8-a4b8-11e3-97c3-866382997e69,0
} joined {
} left {
} partitioned {
})
140305 17:49:42 [Note] WSREP: gcomm: connected
140305 17:49:42 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
140305 17:49:42 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
140305 17:49:42 [Note] WSREP: Opened channel 'controller_cluster'
140305 17:49:42 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
140305 17:49:42 [Note] WSREP: Waiting for SST to complete.
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: sent state msg: 52478b99-a4b8-11e3-9283-8207651d087e
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: got state msg: 52478b99-a4b8-11e3-9283-8207651d087e from 0 (db01.deg.pod1)
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: got state msg: 52478b99-a4b8-11e3-9283-8207651d087e from 1 (db02.deg.pod1)
140305 17:49:42 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 47,
members = 1/2 (joined/total),
act_id = 1,
last_appl. = -1,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = bce8f04b-a41a-11e3-b010-4ba4a408598c
140305 17:49:42 [Note] WSREP: Flow-control interval: [23, 23]
140305 17:49:42 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
140305 17:49:42 [Note] WSREP: State transfer required:
Group state: bce8f04b-a41a-11e3-b010-4ba4a408598c:1
Local state: 00000000-0000-0000-0000-000000000000:-1
140305 17:49:42 [Note] WSREP: New cluster view: global state: bce8f04b-a41a-11e3-b010-4ba4a408598c:1, view# 48: Primary, number of nodes: 2, my index: 1, protocol version 2
140305 17:49:42 [Warning] WSREP: Gap in state sequence. Need state transfer.
140305 17:49:44 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.16.1.21' --auth 'wsrep_sst:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '13639''
140305 17:49:44 [Note] WSREP: Prepared SST request: rsync|172.16.1.21:4444/rsync_sst
140305 17:49:44 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140305 17:49:44 [Note] WSREP: REPL Protocols: 5 (3, 1)
140305 17:49:44 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
140305 17:49:44 [Note] WSREP: Service thread queue flushed.
140305 17:49:44 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bce8f04b-a41a-11e3-b010-4ba4a408598c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():445. IST will be unavailable.
140305 17:49:44 [Note] WSREP: Node 1.0 (db02.deg.pod1) requested state transfer from '*any*'. Selected 0.0 (db01.deg.pod1)(SYNCED) as donor.
140305 17:49:44 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
140305 17:49:44 [Note] WSREP: Requesting state transfer: success, donor: 0
140305 17:49:45 [Warning] WSREP: 0.0 (db01.deg.pod1): State transfer to 1.0 (db02.deg.pod1) failed: -1 (Operation not permitted)
140305 17:49:45 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
140305 17:49:45 [Note] WSREP: gcomm: terminating thread
140305 17:49:45 [Note] WSREP: gcomm: joining thread
140305 17:49:45 [Note] WSREP: gcomm: closing backend
140305 17:49:46 [Note] WSREP: view(view_id(NON_PRIM,3f183cba-a422-11e3-b1c7-52f230abd39f,48) memb {
7036f7c8-a4b8-11e3-97c3-866382997e69,0
} joined {
} left {
} partitioned {
3f183cba-a422-11e3-b1c7-52f230abd39f,0
})
140305 17:49:46 [Note] WSREP: view((empty))
140305 17:49:46 [Note] WSREP: gcomm: closed
140305 17:49:46 [Note] WSREP: /usr/sbin/mysqld: Terminated.
140305 17:49:46 mysqld_safe mysqld from pid file /var/lib/mysql/db02.deg.pod1.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:13639) terminated unexpectedly. (20140305 17:49:47.601)
WSREP_SST: [INFO] Joiner cleanup. (20140305 17:49:47.603)
WSREP_SST: [INFO] Joiner cleanup done. (20140305 17:49:48.110)
One possible reason for nodes failing to add is that the existing nodes don't expect them. In your configuration files, you may just need to point each node to the rest.
For example, try using wsrep_cluster_address="gcomm://"172.16.1.20,172.16.1.21" in both galera.cnf files.