Nodes unexpectedly closing in Mariadb Galera Cluster - mysql

I've setup 3 node galera cluster for my application, there is a certain stored procedure in my application that creates a table temporarily(Not TEMPORARY TABLE), the table is created dynamically and executed. Some CRUD operation is done on the table muptiple times and after its usage is completed it is dropped at the end of the stored procedure.
The problem is that everytime I run this particular stored procedure 2 out of my 3 node fails and shuts down automatically.
Here is the Log from the node that fails :-
05 seqnos (l: 38, g: 47, s: 46, d: 46, ts: 186965868554819)
150423 8:31:35 [ERROR] WSREP: Failed to apply trx 47 4 times
150423 8:31:35 [ERROR] WSREP: Node consistency compromized, aborting...
150423 8:31:35 [Note] WSREP: Closing send monitor...
150423 8:31:35 [Note] WSREP: Closed send monitor.
150423 8:31:35 [Note] WSREP: gcomm: terminating thread
150423 8:31:35 [Note] WSREP: gcomm: joining thread
150423 8:31:35 [Note] WSREP: gcomm: closing backend
150423 8:31:35 [Note] WSREP: view(view_id(NON_PRIM,4d2adf77-e972-11e4-be7e-6bcb3d5c882f,3) memb {
849bbb26-e981-11e4-9f9f-f37ebca03acf,0
} joined {
} left {
} partitioned {
4d2adf77-e972-11e4-be7e-6bcb3d5c882f,0
634f32cb-e972-11e4-a226-bf661daf79d7,0
})
150423 8:31:35 [Note] WSREP: view((empty))
150423 8:31:35 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
150423 8:31:35 [Note] WSREP: gcomm: closed
150423 8:31:35 [Note] WSREP: Flow-control interval: [16, 16]
150423 8:31:35 [Note] WSREP: Received NON-PRIMARY.
150423 8:31:35 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 47)
150423 8:31:35 [Note] WSREP: Received self-leave message.
150423 8:31:35 [Note] WSREP: Flow-control interval: [0, 0]
150423 8:31:35 [Note] WSREP: Received SELF-LEAVE. Closing connection.
150423 8:31:35 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 47)
150423 8:31:35 [Note] WSREP: RECV thread exiting 0: Success
150423 8:31:35 [Note] WSREP: recv_thread() joined.
150423 8:31:35 [Note] WSREP: Closing replication queue.
150423 8:31:35 [Note] WSREP: Closing slave action queue.
150423 8:31:35 [Note] WSREP: /usr/sbin/mysqld: Terminated.
150423 08:31:35 mysqld_safe Number of processes running now: 0
150423 08:31:35 mysqld_safe WSREP: not restarting wsrep node automatically
150423 08:31:35 mysqld_safe mysqld from pid file /mnt/mysql/db3-prod.pid ended
is There any particular I can solve this without modifying the internal structure of the stored procedure.
The output of
SHOW VARIABLES LIKE 'default%'; is
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| default_master_connection | |
| default_regex_flags | |
| default_storage_engine | InnoDB |
| default_week_format | 0 |
+---------------------------+--------+

Related

Mariadb Galera node will not successfully rejoin unless it picks a particular node to rejoin via

Debian 10, Maridadb 10.3.26, Galera-3 25.3.31
I have a three node cluster. The nodes are named node3, node4, and node5. Node3 gets disconnected from the cluster on occasion.
If it picks node5 to recover from I get
2020-11-18 19:42:08 7 [Note] WSREP: Requesting state transfer: success, donor: 2
2020-11-18 19:42:08 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75720363
2020-11-18 19:42:08 17 [Note] WSREP: SST received: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989
2020-11-18 19:42:08 17 [Note] WSREP: wsrep_start_position set to '57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989'
2020-11-18 19:42:08 7 [Note] WSREP: Receiving IST: 23374 writesets, seqnos 75696989-75720363
2020-11-18 19:42:08 0 [Note] WSREP: 2.0 (node5): State transfer to 0.0 (node3) complete.
2020-11-18 19:42:08 0 [Note] WSREP: Member 2.0 (node5) synced with group.
2020-11-18 19:42:08 0 [Note] WSREP: (23249d11, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-11-18 19:42:15 0 [Warning] WSREP: Protocol violation. JOIN message sender 2.0 (node5) is not in state transfer (SYNCED). Message ignored.
after which node3 will sit forever never changing wsrep_ready to yes.
On the other hand, if node3 picks node4 I get all the same sort of messages except
[Warning] WSREP: Protocol violation. JOIN message sender 2.0 (node5) is not in state transfer (SYNCED). Message ignored.
does not appear and eventually node3 wsrep_ready becomes yes and the node starts to process queries.
Any idea how I much figure out the issue?
Here is some more data. This is an example of a successful join when it chooses node4 instead of node5:
2020-11-19 21:12:54 7 [Note] WSREP: State transfer required:
Group state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331
Local state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75696989
2020-11-19 21:12:54 7 [Note] WSREP: REPL Protocols: 9 (4, 2)
2020-11-19 21:12:54 7 [Note] WSREP: New cluster view: global state: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331, view# 349: Primary, number of nodes: 3, my index: 2, protocol version 3
2020-11-19 21:12:54 7 [Warning] WSREP: Gap in state sequence. Need state transfer.
2020-11-19 21:12:56 7 [Note] WSREP: Prepared SST request: mysqldump|10.4.44.82:3360
2020-11-19 21:12:56 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-11-19 21:12:56 7 [Note] WSREP: Assign initial position for certification: 75815331, protocol version: 4
2020-11-19 21:12:56 0 [Note] WSREP: Service thread queue flushed.
2020-11-19 21:12:56 7 [Note] WSREP: IST receiver addr using tcp://x.y.z.a:4568
2020-11-19 21:12:56 7 [Note] WSREP: Prepared IST receiver, listening at: tcp://x.y.z.a:4568
2020-11-19 21:12:56 0 [Note] WSREP: Member 2.0 (node3) requested state transfer from '*any*'. Selected 0.0 (node4)(SYNCED) as donor.
2020-11-19 21:12:56 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 75815331)
2020-11-19 21:12:56 7 [Note] WSREP: Requesting state transfer: success, donor: 0
2020-11-19 21:12:56 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:75815331
2020-11-19 21:12:56 0 [Note] WSREP: (fcbfdc45, 'tcp://0.0.0.0:4567') turning message relay requesting off

mariadb, add 4th galera node failed

I have three node setup and running perfectly for the past months.
Recently I want to add another node in a different location but somehow I keep on getting errors.
At first, I was just following this tutorial (where I setup the first time few months ago) https://www.howtoforge.com/tutorial/how-to-install-and-configure-galera-cluster-on-ubuntu-1604/ I did not start all the nodes again from the beginning, I just has to find the file of /mysql/conf.d/galera.cnf in the other three nodes I added the new nodes ip into the previous three. So for the forth one I had the /etc/mysql/conf.d/galera.cnf setup like...
[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
# Galera Cluster Configuration
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://node1_ip,node2_ip,node3_ip,node4_ip"
# Galera Synchronization Configuration
wsrep_sst_method=rsync
# Galera Node Configuration
wsrep_node_address="xx.xx.xxx.xxx"
wsrep_node_name="Node4"
somehow I am getting this HUGE error,
Group state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
Local state: 00000000-0000-0000-0000-000000000000:-1
[Note] WSREP: New cluster view: global state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273, view# 122: Primary, number of nodes: 4, my
[Warning] WSREP: Gap in state sequence. Need state transfer.
[Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/' --parent '22828' ''
rsyncd version 3.1.1 starting, listening on port 4444
[Note] WSREP: Prepared SST request: rsync|xxx.node.4.ip:4444/rsync_sst
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
[Note] WSREP: REPL Protocols: 7 (3, 2)
[Note] WSREP: Assign initial position for certification: 36273, protocol version: 3
[Note] WSREP: Service thread queue flushed.
[Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
[Note] WSREP: Member 0.0 (Node4) requested state transfer from '*any*'. Selected 1.0 (Node1)(SYNCED) as donor.
[Note] WSREP: Shifting PRIMARY -> JOINER (TO: 36273)
[Note] WSREP: Requesting state transfer: success, donor: 1
[Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
[Note] WSREP: (7642cf37, 'tcp://0.0.0.0:4567') connection to peer 7642cf37 with addr tcp://xxx.node.4.ip:4567 timed out, no messages
[Note] WSREP: (7642cf37, 'tcp://0.0.0.0:4567') turning message relay requesting off
mariadb.service: Start operation timed out. Terminating.
Terminated
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 22875
sent 0 bytes received 0 bytes total size 0
WSREP_SST: [INFO] Joiner cleanup done.
[ERROR] WSREP: Process was aborted.
[ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/'
[ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
[ERROR] WSREP: SST failed: 2 (No such file or directory)
[ERROR] Aborting
Error in my_thread_global_end(): 1 threads didn't exit
mariadb.service: Main process exited, code=exited, status=1/FAILURE
Failed to start MariaDB 10.1.33 database server.
P.S for the older 3 nodes maria db version is 10.1.29 and the new node is 10.1.33
Thanks in advance for any suggestions.

Percona/Galera cluster: Node consistency compromized, aborting

I have a two separate Galera clusters, will call them C1 and C2.
C1 is a main cluster and has a three nodes, C2 is emergency and has only one node. C1 connected with C2 by GTID replication for providing completely identity.
The main problem that I can't increase a number of nodes in C2 cluster. I get these errors:
2016-08-02 12:54:00 19067 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 2th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 3th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 4th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [ERROR] WSREP: Failed to apply trx: source: 377c1c2e-5885-11e6-aed8-1e04b1e53bff version: 3 local: 0 state: APPLYING flags: 1 conn_id: 346 trx
_id: 23137618 seqnos (l: 5, g: 12305593, s: 12305592, d: 12305592, ts: 16848639299414398)
2016-08-02 13:00:58 19067 [ERROR] WSREP: Failed to apply trx 12305593 4 times
2016-08-02 13:00:58 19067 [ERROR] WSREP: Node consistency compromized, aborting...
2016-08-02 13:00:58 19067 [Note] WSREP: Closing send monitor...
2016-08-02 13:00:58 19067 [Note] WSREP: Closed send monitor.
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: terminating thread
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: joining thread
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: closing backend
The main settings that relates to the issue:
C1 and C2:
binlog_format=ROW
wsrep_sst_method=xtrabackup-v2
server_id = 1 (C1); server_id = 101 (C2)
wsrep_OSU_method = 'TOI'
What is most interesting, that with simple async replication (not GTID) all worked properly. Have you any ideas, guys?

MariaDB Galera Cluster set up problems

I am trying to get a mariadb cluster up and running but it is not working out for me. Right now I am using MariaDB Galera 5.5.36 on a 64 bit red hat ES6 machine. I installed mariadb through this repo here:
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5-galera/rhel6-amd64/
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
In the server.conf I have the following in server 1:
[mariadb]
log_error=/var/log/mariadb.log
query_cache_size=0
query_cache_type=0
binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://192.168.211.133
wsrep_cluster_name='cluster'
wsrep_node_address='192.168.211.132'
wsrep_node_name='cluster1'
wsrep_sst_method=rsync
and on server 2 I have
[mariadb]
log_error=/var/log/mariadb.log
query_cache_size=0
query_cache_type=0
binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://192.168.211.132
wsrep_cluster_name='cluster'
wsrep_node_address='192.168.211.133'
wsrep_node_name='cluster2'
wsrep_sst_method=rsync
When I start server 1 with the following command: sudo service mysql start --wsrep-new-cluster it starts up just fine, if I open up mysql and check the status of wsrep it says everything is up and running which is good but when I try to do sudo service mysql start on the second server I get the following in the error logs:
140609 14:47:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140609 14:47:56 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.i5qfm2' --pid-file='/var/lib/mysql/localhost.localdomain-recover.pid'
140609 14:47:57 mysqld_safe WSREP: Recovered position 85448d73-ebe8-11e3-9c20-fbc1995fee11:0
140609 14:47:57 [Note] WSREP: wsrep_start_position var submitted: '85448d73-ebe8-11e3-9c20-fbc1995fee11:0'
140609 14:47:57 [Note] WSREP: Read nil XID from storage engines, skipping position init
140609 14:47:57 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
140609 14:47:57 [Note] WSREP: wsrep_load(): Galera 25.3.2(r170) by Codership Oy <info#codership.com> loaded successfully.
140609 14:47:57 [Note] WSREP: CRC-32C: using hardware acceleration.
140609 14:47:57 [Note] WSREP: Found saved state: 85448d73-ebe8-11e3-9c20-fbc1995fee11:-1
140609 14:47:57 [Note] WSREP: Passing config to GCS: base_host = 192.168.211.133; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5
140609 14:47:57 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
140609 14:47:57 [Note] WSREP: wsrep_sst_grab()
140609 14:47:57 [Note] WSREP: Start replication
140609 14:47:57 [Note] WSREP: Setting initial position to 85448d73-ebe8-11e3-9c20-fbc1995fee11:0
140609 14:47:57 [Note] WSREP: protonet asio version 0
140609 14:47:57 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
140609 14:47:57 [Note] WSREP: backend: asio
140609 14:47:57 [Note] WSREP: GMCast version 0
140609 14:47:57 [Note] WSREP: (0c085f34-efe5-11e3-9f6b-8bfd1706e2a4, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
140609 14:47:57 [Note] WSREP: (0c085f34-efe5-11e3-9f6b-8bfd1706e2a4, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
140609 14:47:57 [Note] WSREP: EVS version 0
140609 14:47:57 [Note] WSREP: PC version 0
140609 14:47:57 [Note] WSREP: gcomm: connecting to group 'cluster', peer '192.168.211.132:,192.168.211.134:'
140609 14:48:00 [Warning] WSREP: no nodes coming from prim view, prim not possible
140609 14:48:00 [Note] WSREP: view(view_id(NON_PRIM,0c085f34-efe5-11e3-9f6b-8bfd1706e2a4,1) memb {
0c085f34-efe5-11e3-9f6b-8bfd1706e2a4,0
} joined {
} left {
} partitioned {
})
140609 14:48:01 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50775S), skipping check
140609 14:48:31 [Note] WSREP: view((empty))
140609 14:48:31 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():141
140609 14:48:31 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():196: Failed to open backend connection: -110 (Connection timed out)
140609 14:48:31 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'cluster' at 'gcomm://192.168.211.132,192.168.211.134': -110 (Connection timed out)
140609 14:48:31 [ERROR] WSREP: gcs connect failed: Connection timed out
140609 14:48:31 [ERROR] WSREP: wsrep::connect() failed: 7
140609 14:48:31 [ERROR] Aborting
140609 14:48:31 [Note] WSREP: Service disconnected.
140609 14:48:32 [Note] WSREP: Some threads may fail to exit.
140609 14:48:32 [Note] /usr/sbin/mysqld: Shutdown complete
140609 14:48:32 mysqld_safe mysqld from pid file /var/lib/mysql/localhost.localdomain.pid ended
I am at a loss as to why the second server cannot detect that a cluster is up and running. These machines can communicate with each other just fine, I can SSH from one to the other and they can ping each other. I tried deleted the galera cache, tried downgrading my version of mariadb galera, tried disabling SELinux, tried running the mysql service as a different user, verified that the correct ports are open, tried running them on 2 VMs on separate computers with different IP addresses, etc. Does anyone have any idea what is going on here because I have been searching for 3 days trying to fix this but no solution seems to work with me.
Here is how I fixed my similar issue.
CentOS 7 w/ MariaDB Galera 10.1.
Node2 I saw this:
016-12-27 15:40:38 140703512762624 [Warning] WSREP: no nodes coming from prim view, prim not possible
After doing some reading, I tried running this on node1.
service mysql start --wsrep-new-cluster
But this failed, and in the logs, I found this...
2016-12-27 15:44:08 140438853814528 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .
So I edited the file /var/lib/mysql/grastate.dat, changing safe_to_bootstrap to 1.
I was then able to start the Primary node using:
service mysql start --wsrep-new-cluster
Then on the others, I just used
service mysql start
Note: This was in a demo pre-production environment. I promptly broke it after getting everything to work by rebooting all servers at the same time :P, but I knew there were no writes, and that the DB's were in sync. If you are in produciton and this happens, you can use the following to figure out which node to run "new-cluster" on, which is akin to saying, make me primary.
mysqld_safe --wsrep-recover
If this is a production issue, I highly reccomend reading this article and making a backup w/ CloneZilla before throwing commands at the broken clients!
https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/
The cluster must start with this command on primary node:
galera_new_cluster
after starting first node, you can start other nodes in the cluster successfully.
I believe you need to list all the IPs in the wsrep_cluster_address parameter.
wsrep_cluster_address=gcomm://192.168.211.132,192.168.211.133
This should be done on both hosts. BTW you likely want three nodes not two as to avoid split brain scenarios.

Additional Nodes on Galera MySQL failing to add

Ok so I have a second node I am trying to add to a working galera mysql server as another node...configs here
Node A(working)
[server]
[mysqld]
[embedded]
[mysqld-5.5]
[mariadb]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=172.16.1.20
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="controller_cluster"
wsrep_cluster_address="gcomm://"
wsrep_sst_receive_addres="172.16.1.20"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
wsrep_sst_auth=wsrep_sst:password
[mariadb-5.5]
Node B(wont start)
[server]
[mysqld]
skip-name-resolve
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
[embedded]
[mysqld-5.5]
[mariadb]
log = /var/log/mysqld.log
log-error = /var/log/mysqld.error.log
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
bind-address=172.16.1.21
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="controller_cluster"
wsrep_cluster_address="gcomm://172.16.1.20"
wsrep_sst_receive_addres="172.16.1.21"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
wsrep_sst_auth=wsrep_sst:password
[mariadb-5.5]
Node A Permissions on /var/lib/mysql
-rw-rw----. 1 mysql mysql 16384 Mar 4 23:54 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Mar 4 23:54 aria_log_control
-rw-r-----. 1 mysql root 283162 Mar 5 17:49 db01.deg.pod1.err
-rw-rw----. 1 mysql mysql 5 Mar 4 23:54 db01.deg.pod1.pid
-rw-------. 1 mysql mysql 134219040 Mar 5 17:48 galera.cache
-rw-rw----. 1 mysql mysql 104 Mar 5 17:48 grastate.dat
-rw-rw----. 1 mysql mysql 12582912 Mar 4 23:54 ibdata1
-rw-rw----. 1 mysql mysql 5242880 Mar 4 23:54 ib_logfile0
-rw-rw----. 1 mysql mysql 5242880 Mar 4 22:30 ib_logfile1
drwx------. 2 mysql mysql 4096 Mar 4 22:59 mysql
srwxrwxrwx. 1 mysql mysql 0 Mar 4 23:54 mysql.sock
drwx------. 2 root root 4096 Mar 4 22:59 performance_schema
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:11 RPM_UPGRADE_HISTORY
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:11 RPM_UPGRADE_MARKER-LAST
drwxr-xr-x. 2 mysql mysql 4096 Mar 4 22:11 test
drwx------. 2 mysql mysql 4096 Mar 5 17:35 tt
Node B Permissions on /var/lib/mysql
-rw-rw----. 1 mysql mysql 16384 Mar 5 17:49 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Mar 5 17:49 aria_log_control
-rw-r-----. 1 mysql root 0 Mar 5 17:49 db02.deg.pod1.err
-rw-------. 1 mysql mysql 134219040 Mar 5 17:49 galera.cache
-rw-rw----. 1 mysql mysql 104 Mar 5 17:49 grastate.dat
-rw-rw----. 1 mysql mysql 12582912 Mar 5 17:49 ibdata1
-rw-rw----. 1 mysql mysql 5242880 Mar 5 17:49 ib_logfile0
-rw-rw----. 1 mysql mysql 5242880 Mar 5 17:49 ib_logfile1
drwx------. 2 mysql mysql 4096 Mar 4 23:10 mysql
srwxrwxrwx 1 mysql mysql 0 Mar 5 17:49 mysql.sock
-rw-------. 1 root root 107 Mar 4 23:10 nohup.out
-rw-r--r-- 1 root root 269455 Mar 5 03:42 out.log
drwx------ 2 root root 4096 Mar 5 03:20 performance_schema
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:14 RPM_UPGRADE_HISTORY
-rw-r--r--. 1 mysql mysql 124 Mar 4 22:14 RPM_UPGRADE_MARKER-LAST
drwxr-xr-x. 2 mysql mysql 4096 Mar 4 22:14 test
drwx------ 2 mysql mysql 4096 Mar 5 17:36 tt
-rw------- 1 root root 0 Mar 5 03:52 wsrep_recovery.hh4i9
Password is also the same on both ends for mysql user and for root user.
** Failure log on Node B **
140305 17:49:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140305 17:49:39 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.A8nOzH' --pid-file='/var/lib/mysql/db02.deg.pod1-recover.pid'
nohup: ignoring input
140305 17:49:39 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:39 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 mysqld_safe WSREP: Recovered position bce8f04b-a41a-11e3-b010-4ba4a408598c:0
140305 17:49:41 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 [Warning] The syntax '--log' is deprecated and will be removed in a future release. Please use '--general-log'/'--general-log-file' instead.
140305 17:49:41 [Note] WSREP: wsrep_start_position var submitted: 'bce8f04b-a41a-11e3-b010-4ba4a408598c:0'
140305 17:49:41 [Note] WSREP: Read nil XID from storage engines, skipping position init
140305 17:49:41 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
140305 17:49:41 [Note] WSREP: wsrep_load(): Galera 25.3.2(r170) by Codership Oy <info#codership.com> loaded successfully.
140305 17:49:41 [Note] WSREP: CRC-32C: using hardware acceleration.
140305 17:49:41 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
140305 17:49:41 [Note] WSREP: Passing config to GCS: base_host = 172.16.1.21; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2147483647; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5
140305 17:49:41 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
140305 17:49:41 [Note] WSREP: wsrep_sst_grab()
140305 17:49:41 [Note] WSREP: Start replication
140305 17:49:41 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
140305 17:49:41 [Note] WSREP: protonet asio version 0
140305 17:49:41 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
140305 17:49:41 [Note] WSREP: backend: asio
140305 17:49:41 [Note] WSREP: GMCast version 0
140305 17:49:41 [Note] WSREP: (7036f7c8-a4b8-11e3-97c3-866382997e69, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
140305 17:49:41 [Note] WSREP: (7036f7c8-a4b8-11e3-97c3-866382997e69, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
140305 17:49:41 [Note] WSREP: EVS version 0
140305 17:49:41 [Note] WSREP: PC version 0
140305 17:49:41 [Note] WSREP: gcomm: connecting to group 'controller_cluster', peer '172.16.1.20:'
140305 17:49:41 [Note] WSREP: declaring 3f183cba-a422-11e3-b1c7-52f230abd39f stable
140305 17:49:41 [Note] WSREP: Node 3f183cba-a422-11e3-b1c7-52f230abd39f state prim
140305 17:49:41 [Note] WSREP: view(view_id(PRIM,3f183cba-a422-11e3-b1c7-52f230abd39f,48) memb {
3f183cba-a422-11e3-b1c7-52f230abd39f,0
7036f7c8-a4b8-11e3-97c3-866382997e69,0
} joined {
} left {
} partitioned {
})
140305 17:49:42 [Note] WSREP: gcomm: connected
140305 17:49:42 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
140305 17:49:42 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
140305 17:49:42 [Note] WSREP: Opened channel 'controller_cluster'
140305 17:49:42 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
140305 17:49:42 [Note] WSREP: Waiting for SST to complete.
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: sent state msg: 52478b99-a4b8-11e3-9283-8207651d087e
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: got state msg: 52478b99-a4b8-11e3-9283-8207651d087e from 0 (db01.deg.pod1)
140305 17:49:42 [Note] WSREP: STATE EXCHANGE: got state msg: 52478b99-a4b8-11e3-9283-8207651d087e from 1 (db02.deg.pod1)
140305 17:49:42 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 47,
members = 1/2 (joined/total),
act_id = 1,
last_appl. = -1,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = bce8f04b-a41a-11e3-b010-4ba4a408598c
140305 17:49:42 [Note] WSREP: Flow-control interval: [23, 23]
140305 17:49:42 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
140305 17:49:42 [Note] WSREP: State transfer required:
Group state: bce8f04b-a41a-11e3-b010-4ba4a408598c:1
Local state: 00000000-0000-0000-0000-000000000000:-1
140305 17:49:42 [Note] WSREP: New cluster view: global state: bce8f04b-a41a-11e3-b010-4ba4a408598c:1, view# 48: Primary, number of nodes: 2, my index: 1, protocol version 2
140305 17:49:42 [Warning] WSREP: Gap in state sequence. Need state transfer.
140305 17:49:44 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.16.1.21' --auth 'wsrep_sst:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '13639''
140305 17:49:44 [Note] WSREP: Prepared SST request: rsync|172.16.1.21:4444/rsync_sst
140305 17:49:44 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140305 17:49:44 [Note] WSREP: REPL Protocols: 5 (3, 1)
140305 17:49:44 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
140305 17:49:44 [Note] WSREP: Service thread queue flushed.
140305 17:49:44 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bce8f04b-a41a-11e3-b010-4ba4a408598c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():445. IST will be unavailable.
140305 17:49:44 [Note] WSREP: Node 1.0 (db02.deg.pod1) requested state transfer from '*any*'. Selected 0.0 (db01.deg.pod1)(SYNCED) as donor.
140305 17:49:44 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
140305 17:49:44 [Note] WSREP: Requesting state transfer: success, donor: 0
140305 17:49:45 [Warning] WSREP: 0.0 (db01.deg.pod1): State transfer to 1.0 (db02.deg.pod1) failed: -1 (Operation not permitted)
140305 17:49:45 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
140305 17:49:45 [Note] WSREP: gcomm: terminating thread
140305 17:49:45 [Note] WSREP: gcomm: joining thread
140305 17:49:45 [Note] WSREP: gcomm: closing backend
140305 17:49:46 [Note] WSREP: view(view_id(NON_PRIM,3f183cba-a422-11e3-b1c7-52f230abd39f,48) memb {
7036f7c8-a4b8-11e3-97c3-866382997e69,0
} joined {
} left {
} partitioned {
3f183cba-a422-11e3-b1c7-52f230abd39f,0
})
140305 17:49:46 [Note] WSREP: view((empty))
140305 17:49:46 [Note] WSREP: gcomm: closed
140305 17:49:46 [Note] WSREP: /usr/sbin/mysqld: Terminated.
140305 17:49:46 mysqld_safe mysqld from pid file /var/lib/mysql/db02.deg.pod1.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:13639) terminated unexpectedly. (20140305 17:49:47.601)
WSREP_SST: [INFO] Joiner cleanup. (20140305 17:49:47.603)
WSREP_SST: [INFO] Joiner cleanup done. (20140305 17:49:48.110)
One possible reason for nodes failing to add is that the existing nodes don't expect them. In your configuration files, you may just need to point each node to the rest.
For example, try using wsrep_cluster_address="gcomm://"172.16.1.20,172.16.1.21" in both galera.cnf files.