Percona xtradb cluster a node can't join server - mysql

Question

My node can't join the cluster.
I'm installed both servers in Linux Debian 10, the server bootsrap ok, but the node can't join the server.
I tried several settings and follow the guide, but without success.
My server configuration is this:
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock
ssl-ca=/etc/mysql/certs/ca.pem
ssl-cert=/etc/mysql/certs/client-cert.pem
ssl-key=/etc/mysql/certs/client-key.pem
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
ssl-ca=/etc/mysql/certs/ca.pem
ssl-cert=/etc/mysql/certs/server-cert.pem
ssl-key=/etc/mysql/certs/server-key.pem
lower_case_table_names=1
#innodb_dedicated_server=1
innodb_buffer_pool_size=2500M
innodb_log_buffer_size=16M
#expire_logs_days = 3
#max_binlog_size = 300M
#default-storage-engine = InnoDB
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
#wsrep_cluster_address=gcomm://
wsrep_cluster_address=gcomm://152.44.34.207,3.143.85.192
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
wsrep_node_address=152.44.34.207
# Cluster name
wsrep_cluster_name=pxc-cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=pxc-cluster-node-1
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem
My node configuration, this:
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock
ssl-ca=/etc/mysql/certs/ca.pem
ssl-cert=/etc/mysql/certs/client-cert.pem
ssl-key=/etc/mysql/certs/client-key.pem
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
ssl-ca=/etc/mysql/certs/ca.pem
ssl-cert=/etc/mysql/certs/server-cert.pem
ssl-key=/etc/mysql/certs/server-key.pem
lower_case_table_names=1
#innodb_dedicated_server=1
innodb_buffer_pool_size=2500M
innodb_log_buffer_size=16M
#expire_logs_days = 3
#max_binlog_size = 300M
#default-storage-engine = InnoDB
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
#wsrep_cluster_address=gcomm://
wsrep_cluster_address=gcomm://152.44.34.207,3.143.85.192
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
wsrep_node_address=3.143.85.192
# Cluster name
wsrep_cluster_name=pxc-cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=pxc-cluster-node-2
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem
The server error log is:
2021-06-22T13:53:46.793403Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://3.143.85.192:42862 local endpoint ssl://152.44.34.207:4567 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-06-22T13:53:46.815265Z 0 [Note] [MY-000000] [Galera] (34eb80d9-8ec1, 'ssl://0.0.0.0:4567') connection established to 43c3db4f-8b6f ssl://3.143.85.192:4567
2021-06-22T13:53:46.836932Z 0 [Note] [MY-000000] [Galera] (34eb80d9-8ec1, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2021-06-22T13:53:47.289191Z 0 [Note] [MY-000000] [Galera] declaring 43c3db4f-8b6f at ssl://3.143.85.192:4567 stable
2021-06-22T13:53:47.310756Z 0 [Note] [MY-000000] [Galera] Node 34eb80d9-8ec1 state primary
2021-06-22T13:53:47.332131Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,34eb80d9-8ec1,2)
memb {
34eb80d9-8ec1,0
43c3db4f-8b6f,0
}
joined {
}
left {
}
partitioned {
}
)
2021-06-22T13:53:47.332202Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2021-06-22T13:53:47.335049Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2021-06-22T13:53:47.335191Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 4422c892-d361-11eb-91ba-5a82f4f6395e
2021-06-22T13:53:47.360122Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 4422c892-d361-11eb-91ba-5a82f4f6395e
2021-06-22T13:53:47.381654Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 4422c892-d361-11eb-91ba-5a82f4f6395e from 0 (pxc-cluster-node-1)
2021-06-22T13:53:47.768196Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 4422c892-d361-11eb-91ba-5a82f4f6395e from 1 (pxc-cluster-node-2)
2021-06-22T13:53:47.768277Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 33,
last_appl. = 32,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe
2021-06-22T13:53:47.768356Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2021-06-22T13:53:47.768461Z 2 [Note] [MY-000000] [Galera] ####### processing CC 34, local, ordered
2021-06-22T13:53:47.768494Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 33 upto current CC event 34 upto:33
2021-06-22T13:53:47.768506Z 2 [Note] [MY-000000] [Galera] Drain monitors from 33 up to 33
2021-06-22T13:53:47.768516Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 34eb80d9-d361-11eb-8ec1-5a3b0be2fce9
2021-06-22T13:53:47.768524Z 2 [Note] [MY-000000] [Galera] Skipping cert index reset
2021-06-22T13:53:47.768531Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2021-06-22T13:53:47.768540Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 33 -> 34
2021-06-22T13:53:47.768580Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-06-22T13:53:47.771208Z 2 [Note] [MY-000000] [Galera] ================================================
View:
id: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe:34
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(2):
0: 34eb80d9-d361-11eb-8ec1-5a3b0be2fce9, pxc-cluster-node-1
1: 43c3db4f-d361-11eb-8b6f-3e5ca8791b18, pxc-cluster-node-2
=================================================
2021-06-22T13:53:47.771239Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-06-22T13:53:47.775065Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 34
2021-06-22T13:53:47.775089Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 34
2021-06-22T13:53:47.775098Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2021-06-22T13:53:48.811770Z 0 [Note] [MY-000000] [Galera] Member 1.0 (pxc-cluster-node-2) requested state transfer from '*any*'. Selected 0.0 (pxc-cluster-node-1)(SYNCED) as donor.
2021-06-22T13:53:48.811843Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> DONOR/DESYNCED (TO: 34)
2021-06-22T13:53:48.811888Z 2 [Note] [MY-000000] [Galera] Detected STR version: 1, req_len: 64, req: STRv1
2021-06-22T13:53:48.811912Z 2 [Warning] [MY-000000] [Galera] Joiner didn't provide IST connection info - cert. index preload impossible, bailing out.
2021-06-22T13:53:48.833331Z 0 [Warning] [MY-000000] [Galera] 0.0 (pxc-cluster-node-1): State transfer to 1.0 (pxc-cluster-node-2) failed: -42 (No message of desired type)
2021-06-22T13:53:48.833367Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 34)
2021-06-22T13:53:48.854846Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc-cluster-node-1) synced with group.
2021-06-22T13:53:48.854899Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 34)
2021-06-22T13:53:48.854968Z 16 [Note] [MY-000000] [Galera] Server pxc-cluster-node-1 synced with group
2021-06-22T13:53:49.855435Z 0 [Note] [MY-000000] [Galera] forgetting 43c3db4f-8b6f (ssl://3.143.85.192:4567)
2021-06-22T13:53:49.855498Z 0 [Note] [MY-000000] [Galera] (34eb80d9-8ec1, 'ssl://0.0.0.0:4567') turning message relay requesting off
2021-06-22T13:53:49.855528Z 0 [Note] [MY-000000] [Galera] Node 34eb80d9-8ec1 state primary
2021-06-22T13:53:49.855563Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,34eb80d9-8ec1,3)
memb {
34eb80d9-8ec1,0
}
joined {
}
left {
}
partitioned {
43c3db4f-8b6f,0
}
)
2021-06-22T13:53:49.855575Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2021-06-22T13:53:49.857453Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2021-06-22T13:53:49.857546Z 0 [Note] [MY-000000] [Galera] forgetting 43c3db4f-8b6f (ssl://3.143.85.192:4567)
2021-06-22T13:53:49.857598Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 45a3abe8-d361-11eb-a845-5e19d7621cf0
2021-06-22T13:53:49.857618Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 45a3abe8-d361-11eb-a845-5e19d7621cf0
2021-06-22T13:53:49.857634Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 45a3abe8-d361-11eb-a845-5e19d7621cf0 from 0 (pxc-cluster-node-1)
2021-06-22T13:53:49.857663Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 2,
members = 1/1 (primary/total),
act_id = 34,
last_appl. = 32,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe
2021-06-22T13:53:49.857699Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2021-06-22T13:53:49.857801Z 12 [Note] [MY-000000] [Galera] ####### processing CC 35, local, ordered
2021-06-22T13:53:49.857835Z 12 [Note] [MY-000000] [Galera] Maybe drain monitors from 34 upto current CC event 35 upto:34
2021-06-22T13:53:49.857845Z 12 [Note] [MY-000000] [Galera] Drain monitors from 34 up to 34
2021-06-22T13:53:49.857854Z 12 [Note] [MY-000000] [Galera] ####### My UUID: 34eb80d9-d361-11eb-8ec1-5a3b0be2fce9
2021-06-22T13:53:49.857862Z 12 [Note] [MY-000000] [Galera] Skipping cert index reset
2021-06-22T13:53:49.857869Z 12 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5)
2021-06-22T13:53:49.857877Z 12 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 34 -> 35
2021-06-22T13:53:49.857940Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-06-22T13:53:49.859286Z 12 [Note] [MY-000000] [Galera] ================================================
View:
id: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe:35
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: 34eb80d9-d361-11eb-8ec1-5a3b0be2fce9, pxc-cluster-node-1
=================================================
2021-06-22T13:53:49.859323Z 12 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-06-22T13:53:49.862140Z 12 [Note] [MY-000000] [Galera] Recording CC from group: 35
2021-06-22T13:53:49.862164Z 12 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 35
2021-06-22T13:53:49.862173Z 12 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2021-06-22T13:53:55.316854Z 0 [Note] [MY-000000] [Galera] cleaning up 43c3db4f-8b6f (ssl://3.143.85.192:4567)
The node log is:
2021-06-22T13:53:46.446447Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.23-14.1) starting as process 28549
2021-06-22T13:53:46.447416Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2021-06-22T13:53:46.447428Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2021-06-22T13:53:46.450889Z 0 [Warning] [MY-010068] [Server] CA certificate /etc/mysql/certs/ca.pem is self signed.
2021-06-22T13:53:46.450949Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-06-22T13:53:46.450985Z 0 [Note] [MY-000000] [WSREP] New joining cluster node configured to use specified SSL artifacts
2021-06-22T13:53:46.451039Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: 69c901dc-d2e9-11eb-ac90-5ec816a285fd:1
2021-06-22T13:53:46.451084Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib/galera4/libgalera_smm.so'
2021-06-22T13:53:46.451707Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.7(r752664d) by Codership Oy <info#codership.com> loaded successfully.
2021-06-22T13:53:46.451773Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2021-06-22T13:53:46.452068Z 0 [ERROR] [MY-000000] [Galera] Unrecognized parameter '”socket.ssl_key'
2021-06-22T13:53:46.452458Z 0 [Note] [MY-000000] [Galera] Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2021-06-22T13:53:46.452532Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe
Seqno: -1 - -1
Offset: -1
Synced: 0
2021-06-22T13:53:46.452548Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe, offset: -1
2021-06-22T13:53:46.452610Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
2021-06-22T13:53:46.503384Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2021-06-22T13:53:46.503424Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: didn't recover any events.
2021-06-22T13:53:46.504534Z 0 [Note] [MY-000000] [Galera] Complete reset of the galera cache
2021-06-22T13:53:46.584064Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...
2021-06-22T13:53:46.703025Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 3.143.85.192; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl_ca = /etc/mysql/certs/ca.pem; socket.ssl_cert = /etc/mysql/certs/server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = /etc/mysql/certs/server-key.pem;
2021-06-22T13:53:46.712223Z 0 [Note] [MY-000000] [WSREP] Starting replication
2021-06-22T13:53:46.712273Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0
2021-06-22T13:53:46.712294Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2021-06-22T13:53:46.712356Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2021-06-22T13:53:46.712513Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2021-06-22T13:53:46.712534Z 0 [Note] [MY-000000] [Galera] initializing ssl context
2021-06-22T13:53:46.712764Z 0 [Note] [MY-000000] [Galera] backend: asio
2021-06-22T13:53:46.712847Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2021-06-22T13:53:46.712932Z 0 [Warning] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2021-06-22T13:53:46.712949Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2021-06-22T13:53:46.713090Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2021-06-22T13:53:46.713214Z 0 [Note] [MY-000000] [Galera] (43c3db4f-8b6f, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2021-06-22T13:53:46.713228Z 0 [Note] [MY-000000] [Galera] (43c3db4f-8b6f, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2021-06-22T13:53:46.713509Z 0 [Note] [MY-000000] [Galera] EVS version 1
2021-06-22T13:53:46.713588Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'pxc-cluster', peer '152.44.34.207:,3.143.85.192:'
2021-06-22T13:53:46.761126Z 0 [Note] [MY-000000] [Galera] SSL handshake successful, remote endpoint ssl://152.44.34.207:4567 local endpoint ssl://172.31.36.42:42862 cipher: TLS_AES_256_GCM_SHA384 compression: none
2021-06-22T13:53:46.805155Z 0 [Note] [MY-000000] [Galera] (43c3db4f-8b6f, 'ssl://0.0.0.0:4567') connection established to 34eb80d9-8ec1 ssl://152.44.34.207:4567
2021-06-22T13:53:46.805259Z 0 [Note] [MY-000000] [Galera] (43c3db4f-8b6f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2021-06-22T13:53:47.278818Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2021-06-22T13:53:47.278864Z 0 [Note] [MY-000000] [Galera] declaring 34eb80d9-8ec1 at ssl://152.44.34.207:4567 stable
2021-06-22T13:53:47.278886Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2021-06-22T13:53:47.300343Z 0 [Note] [MY-000000] [Galera] Node 34eb80d9-8ec1 state primary
2021-06-22T13:53:47.324984Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,34eb80d9-8ec1,2)
memb {
34eb80d9-8ec1,0
43c3db4f-8b6f,0
}
joined {
}
left {
}
partitioned {
}
)
2021-06-22T13:53:47.325016Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2021-06-22T13:53:47.328229Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://3.143.85.192:4567
2021-06-22T13:53:47.328250Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x55cd9e76eb50
2021-06-22T13:53:47.714199Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2021-06-22T13:53:47.714266Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2021-06-22T13:53:47.714442Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2021-06-22T13:53:47.714463Z 0 [Note] [MY-000000] [Galera] Opened channel 'pxc-cluster'
2021-06-22T13:53:47.714585Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2021-06-22T13:53:47.714664Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2021-06-22T13:53:47.714844Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2021-06-22T13:53:47.714890Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 4422c892-d361-11eb-91ba-5a82f4f6395e
2021-06-22T13:53:47.714925Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 4422c892-d361-11eb-91ba-5a82f4f6395e from 0 (pxc-cluster-node-1)
2021-06-22T13:53:47.715009Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2021-06-22T13:53:47.736445Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 4422c892-d361-11eb-91ba-5a82f4f6395e from 1 (pxc-cluster-node-2)
2021-06-22T13:53:47.736496Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 1,
members = 1/2 (primary/total),
act_id = 33,
last_appl. = 32,
protocols = 2/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe
2021-06-22T13:53:47.736546Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2021-06-22T13:53:47.736560Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 34)
2021-06-22T13:53:47.736621Z 2 [Note] [MY-000000] [Galera] ####### processing CC 34, local, ordered
2021-06-22T13:53:47.736662Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 34 upto:-1
2021-06-22T13:53:47.736676Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2021-06-22T13:53:47.736691Z 2 [Note] [MY-000000] [Galera] Process first view: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe my uuid: 43c3db4f-d361-11eb-8b6f-3e5ca8791b18
2021-06-22T13:53:47.736717Z 2 [Note] [MY-000000] [Galera] Server pxc-cluster-node-2 connected to cluster at position 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe:34 with ID 43c3db4f-d361-11eb-8b6f-3e5ca8791b18
2021-06-22T13:53:47.736733Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2021-06-22T13:53:47.736753Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-06-22T13:53:47.736788Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 43c3db4f-d361-11eb-8b6f-3e5ca8791b18
2021-06-22T13:53:47.736807Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2021-06-22T13:53:47.736881Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2021-06-22T13:53:47.736925Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2021-06-22T13:53:47.736943Z 2 [Note] [MY-000000] [Galera] State transfer required:
Group state: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe:34
Local state: 00000000-0000-0000-0000-000000000000:-1
2021-06-22T13:53:47.736954Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2021-06-22T13:53:47.736965Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2021-06-22T13:53:47.737466Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '3.143.85.192' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '28549' --mysqld-version '8.0.23-14.1' '' )
2021-06-22T13:53:48.388054Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2021-06-22T13:53:48.757877Z 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|3.143.85.192:4444/xtrabackup_sst//1
2021-06-22T13:53:48.757935Z 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2021-06-22T13:53:48.757963Z 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe
2021-06-22T13:53:48.757986Z 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 34, STRv: 3
2021-06-22T13:53:48.758062Z 2 [Note] [MY-000000] [Galera] IST receiver addr using ssl://3.143.85.192:4568
2021-06-22T13:53:48.758114Z 2 [Note] [MY-000000] [Galera] IST receiver using ssl
2021-06-22T13:53:48.758421Z 2 [Note] [MY-000000] [Galera] State gap can't be serviced using IST. Switching to SST
2021-06-22T13:53:48.758438Z 2 [Warning] [MY-000000] [Galera] Failed to prepare for incremental state transfer: Failed to open IST listener at ssl://3.143.85.192:4568', asio error 'bind: Cannot assign requested address': 99 (Cannot assign requested address)
at galera/src/ist.cpp:prepare():376. IST will be unavailable.
2021-06-22T13:53:48.780145Z 0 [Note] [MY-000000] [Galera] Member 1.0 (pxc-cluster-node-2) requested state transfer from '*any*'. Selected 0.0 (pxc-cluster-node-1)(SYNCED) as donor.
2021-06-22T13:53:48.780196Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 34)
2021-06-22T13:53:48.780246Z 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2021-06-22T13:53:48.780273Z 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2021-06-22T13:53:48.780298Z 2 [Note] [MY-000000] [Galera] GCache history reset: 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe:0 -> 45dca0ba-d2cf-11eb-bec1-7eba7f7a7dfe:34
2021-06-22T13:53:48.823302Z 0 [Warning] [MY-000000] [Galera] 0.0 (pxc-cluster-node-1): State transfer to 1.0 (pxc-cluster-node-2) failed: -42 (No message of desired type)
2021-06-22T13:53:48.823350Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1214: Will never receive state. Need to abort.
2021-06-22T13:53:48.823373Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread
2021-06-22T13:53:48.823410Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread
2021-06-22T13:53:48.823481Z 0 [Note] [MY-000000] [Galera] gcomm: closing backend
2021-06-22T13:53:49.847630Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,34eb80d9-8ec1,2)
memb {
43c3db4f-8b6f,0
}
joined {
}
left {
}
partitioned {
34eb80d9-8ec1,0
}
)
2021-06-22T13:53:49.847692Z 0 [Note] [MY-000000] [Galera] (43c3db4f-8b6f, 'ssl://0.0.0.0:4567') turning message relay requesting off
2021-06-22T13:53:49.847714Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2021-06-22T13:53:49.847728Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2021-06-22T13:53:49.847917Z 0 [Note] [MY-000000] [Galera] gcomm: closed
2021-06-22T13:53:49.847939Z 0 [Note] [MY-000000] [Galera] /usr/sbin/mysqld: Terminated.
2021-06-22T13:53:49.847964Z 0 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2021-06-22T13:53:49.847975Z 0 [Note] [MY-000000] [WSREP] Terminating SST process