I have a mysql cluster with 4 data nodes. I am trying to do an insert of 8 million rows, using load data infile in chunks of 50k to keep transactions small. Like clockwork, after every 1 million rows inserted into the table, the insert stalls for nearly 20 minutes before starting again. I also get this error in the logs.
2014-07-17 22:34:59 [NdbApi] ERROR -- Sending TCROLLBACKREQ with Bad flag
I'm assuming I am in need of a configuration change
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=90G # How much memory to allocate for data storage
IndexMemory=10G # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
NoOfFragmentLogFiles=20
MaxNoOfOrderedIndexes=1000
MaxNoOfAttributes=10000
MaxBufferedEpochs=10000
MaxNoOfConcurrentTransactions=1000000
MaxNoOfConcurrentOperations=10000000
MaxNoOfLocalOperations=11000000
[tcp default]
# TCP/IP options:
#portnumber=2202 # This the default; however, you can use any
# port that is free for all the hosts in the cluster
# Note: It is recommended that you do not specify the port
# number at all and simply allow the default value to be used
# instead
[ndb_mgmd]
# Management process options:
hostname=########.173 # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
nodeId=1
[ndbd]
# Options for data node "A":
hostname=########.173 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
ServerPort=50501
nodeId=2
MaxNoOfExecutionThreads=4
[ndbd]
# Options for data node "B":
hostname=########.174 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
ServerPort=50502
nodeId=3
MaxNoOfExecutionThreads=4
[ndbd]
# Options for data node "C":
hostname=########.175 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
ServerPort=50503
nodeId=5
MaxNoOfExecutionThreads=4
[ndbd]
# Options for data node "D":
hostname=########.176 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
ServerPort=50504
nodeId=6
MaxNoOfExecutionThreads=4
#[ndbd]
# Options for data node "E":
#hostname=########.177 # Hostname or IP address
#datadir=/usr/local/mysql/data # Directory for this data node's data files
#ServerPort=50505
#nodeId=7
#MaxNoOfExecutionThreads=4
[mysqld]
# SQL node options:
nodeId=4
hostname=########.174 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=7
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=8
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=9
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=10
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=11
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=12
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=13
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
[mysqld]
# SQL node options:
nodeId=14
hostname=########.177 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
Changing my configuration to this solved the problem
[TCP DEFAULT]
SendBufferMemory=8M
ReceiveBufferMemory=8M
[NDB_MGMD DEFAULT]
PortNumber=1186
Datadir=/data/mysqlcluster/
[NDB_MGMD]
NodeId=1
Hostname=vacnode6.mc.vanderbilt.edu
LogDestination=FILE:filename=ndb_1_cluster.log,maxsize=10000000,maxfiles=6
ArbitrationRank=1
[NDBD DEFAULT]
NoOfReplicas=2
ServerPort=2200
Datadir=/data/mysqlcluster/
FileSystemPathDD=/data/mysqlcluster/
BackupDataDir=/data/mysqlcluster//backup/
#FileSystemPathUndoFiles=/data/mysqlcluster/
#FileSystemPathDataFiles=/data/mysqlcluster/
DataMemory=95695M
IndexMemory=11962M
LockPagesInMainMemory=1
MaxNoOfConcurrentOperations=100000
MaxNoOfConcurrentTransactions=16384
StringMemory=25
MaxNoOfTables=4096
MaxNoOfOrderedIndexes=2048
MaxNoOfUniqueHashIndexes=512
MaxNoOfAttributes=24576
MaxNoOfTriggers=14336
### Params for REDO LOG
FragmentLogFileSize=256M
InitFragmentLogFiles=SPARSE
NoOfFragmentLogFiles=374
RedoBuffer=64M
TransactionBufferMemory=8M
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=100
TimeBetweenEpochsTimeout=0
### Params for LCP
DiskCheckpointSpeedInRestart=100M
DiskCheckpointSpeed=10M
TimeBetweenLocalCheckpoints=20
### Heartbeating
HeartbeatIntervalDbDb=15000
HeartbeatIntervalDbApi=15000
### Params for setting logging
MemReportFrequency=30
BackupReportFrequency=10
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15
### Params for BACKUP
BackupMaxWriteSize=1M
BackupDataBufferSize=24M
BackupLogBufferSize=16M
BackupMemory=40M
### Params for ODIRECT
#Reports indicates that odirect=1 can cause io errors (os err code 5) on some systems. You must test.
#ODirect=1
### Watchdog
TimeBetweenWatchdogCheckInitial=60000
### TransactionInactiveTimeout - should be enabled in Production
TransactionInactiveTimeout=60000
### New 7.1.10 redo logging parameters
RedoOverCommitCounter=3
RedoOverCommitLimit=20
### REALTIME EXTENSIONS
#RealTimeScheduler=1
### REALTIME EXTENSIONS FOR 6.3 ONLY
#SchedulerExecutionTimer=80
#SchedulerSpinTimer=40
### DISK DATA
SharedGlobalMemory=20M
DiskPageBufferMemory=64M
### Multithreading
MaxNoOfExecutionThreads=8
NoOfFragmentLogParts=4
### Increasing the LongMessageBuffer b/c of a bug (20090903)
LongMessageBuffer=32M
BatchSizePerLocalScan=512
[NDBD]
NodeId=2
Hostname=vacnode1.mc.vanderbilt.edu
### REALTIME ASPECTS - THREAD BINDING
### READ http://johanandersson.blogspot.com/2008/02/mysql-cluster-features-what-they-are.html#realtime
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[NDBD]
NodeId=3
Hostname=vacnode2.mc.vanderbilt.edu
### REALTIME ASPECTS - THREAD BINDING
### READ http://johanandersson.blogspot.com/2008/02/mysql-cluster-features-what-they-are.html#realtime
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[NDBD]
NodeId=4
Hostname=vacnode3.mc.vanderbilt.edu
### REALTIME ASPECTS - THREAD BINDING
### READ http://johanandersson.blogspot.com/2008/02/mysql-cluster-features-what-they-are.html#realtime
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[NDBD]
NodeId=5
Hostname=vacnode4.mc.vanderbilt.edu
### REALTIME ASPECTS - THREAD BINDING
### READ http://johanandersson.blogspot.com/2008/02/mysql-cluster-features-what-they-are.html#realtime
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[MYSQLD DEFAULT]
DefaultOperationRedoProblemAction=QUEUE
BatchSize=512
#BatchByteSize=2048K
#MaxScanBatchSize=2048K
[MYSQLD]
NodeId=6
[MYSQLD]
NodeId=7
[MYSQLD]
NodeId=8
[MYSQLD]
NodeId=9
[MYSQLD]
NodeId=10
[MYSQLD]
NodeId=11
[MYSQLD]
NodeId=12
[MYSQLD]
NodeId=13
[MYSQLD]
NodeId=14
[MYSQLD]
NodeId=15
[MYSQLD]
NodeId=16
[MYSQLD]
NodeId=17
[MYSQLD]
NodeId=18
[MYSQLD]
NodeId=19
[MYSQLD]
NodeId=20
[MYSQLD]
NodeId=21
[MYSQLD]
NodeId=22
[MYSQLD]
NodeId=23
[MYSQLD]
NodeId=24
[MYSQLD]
NodeId=25
[MYSQLD]
NodeId=26
[MYSQLD]
NodeId=27
[MYSQLD]
NodeId=28
[MYSQLD]
NodeId=29
[MYSQLD]
NodeId=30
[MYSQLD]
NodeId=31
[MYSQLD]
NodeId=32
[MYSQLD]
NodeId=33
[MYSQLD]
NodeId=34
[MYSQLD]
NodeId=35
[MYSQLD]
NodeId=36
[MYSQLD]
NodeId=37
[MYSQLD]
[MYSQLD]
[MYSQLD]
[MYSQLD]
### SLOTS (one for each ndb_mgmd) FOR HELPER APPLICATIONS SUCH AS ndb_show_tables etc
[MYSQLD]
Hostname=vacnode6.mc.vanderbilt.edu
### SLOT FOR CMON
[MYSQLD]
Hostname=something.mc.vanderbilt.edu
Related
I have 2 nodes and name in hosts is percona1, percona2 .I have successfully installed percona cluster on both nodes. then i started doing config 2 node , i edited the config file my.cnf
# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/lib/mysql/mysql.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://percona1,percona2
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
wsrep_node_address=percona1
# Cluster name
wsrep_cluster_name=my_centos_cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=cluster_node_1
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING
# SST method
wsrep_sst_method=xtrabackup-v2
After the change work was done I started mysql and got an error , I tried to check every space and got error
enter image description here
Host1:
ip: 42.a1.b1.c1 (in configuration file, a1,b1,c1 will be replaced by real values.)
mysql-cluster-community-server, mysql-cluster-community-management-server and mysql-cluster-community-data-node are all installed on this host.
Host2:
ip: 119.a2.b2.c2 (in configuration file, a2,b2,c2 will be replaced by real values.)
Both mysql-cluster-community-server and mysql-cluster-community-data-node are installed on this host.
All mysql-cluster-* softwares on both hosts didn't start.
cat /var/lib/mysql-cluster/config.ini on host1 outputs:
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=1 # Number of fragment replicas
DataMemory=80M # How much memory to allocate for data storage
IndexMemory=18M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example NDB Cluster setup.
# NOTE: IndexMemory is deprecated in NDB 7.6 and later; in
# these versions, resources for all data and indexes are
# allocated by DataMemory and any that are set for IndexMemory
# are added to the DataMemory resource pool
ServerPort=2202 # This the default value; however, you can use any
# port that is free for all the hosts in the cluster
# Note1: It is recommended that you do not specify the port
# number at all and simply allow the default value to be used
# instead
# Note2: The port was formerly specified using the PortNumber
# TCP parameter; this parameter is no longer available in NDB
# Cluster 7.5.
[ndb_mgmd]
# Management process options:
HostName=42.a1.b1.c1 # Hostname or IP address of management node
DataDir=/var/lib/mysql-cluster # Directory for management node log files
#[ndbd]
#Options for data node "A":
# (one [ndbd] section per data node)
#HostName=42.a1.b1.c1 # Hostname or IP address
#NodeId=2 # Node ID for this data node
#DataDir=/usr/local/mysql/data # Directory for this data node's data files
[ndbd]
#Options for data node "B":
HostName=119.a2.b2.c2 # Hostname or IP address
NodeId=3 # Node ID for this data node
DataDir=/usr/local/mysql/data # Directory for this data node's data files
[mysqld]
#SQL node options:
HostName=119.a2.b2.c2 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --ndb-nodeid=1 on host1 outpus:
MySQL Cluster Management Server mysql-5.7.32 ndb-7.6.16
2020-11-07 16:35:20 [MgmtSrvr] WARNING -- at line 5: [DB] IndexMemory is deprecated, will use Number bytes on each ndbd(DB) node allocated for storing indexes instead
2020-11-07 16:35:20 [MgmtSrvr] ERROR -- The hostname this node should have according to the configuration does not match a local interface. Attempt to bind '42.a1.b1.c1' failed with error: 99 'Cannot assign requested address'
why did the error "The hostname this node should have according to the configuration does not match a local interface" occur?
thanks a lot.
Happen because /etc/hosts has not been loaded in ram at boot time when your ndb service load.
i do resolve this on my case just adding 5 sec pause into the service script.
detail:
[Service]
ExecStartPre=-/bin/sleep 5
I've been trying with intermittent success to restore my ndb_backups to a new cluster.
We have a 6 NDB node cluster with 3 API nodes. When I run an ndb_restore, usually the first 2 or 3 node backups get pulled in without issue. But the 4th and 5th node fail with the following error:
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Retried transaction 10 times.
Last error266: Time-out in NDB, probably caused by deadlock
...Unable to recover from errors. Exiting...
Strangely, sometimes I can simply rerun all 6 backups and it finishes successfully.
I'm hoping someone knows what kind of tweaks I can make to my configuration to optimize this process. Here are my version numbers and configs:
mysql-5.6.22 ndb-7.3.8
My mgm configuration file:
######################
#MGM CONFIG
######################
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=8144M # How much memory to allocate for data storage
IndexMemory=8144M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
[ndb_mgmd]
# Management process options:
hostname=192.168.207.133 # Hostname or IP address of MGM node
NodeId=1
[ndb_mgmd]
# Management process options:
hostname=192.168.207.45 # Hostname or IP address of MGM node
NodeId=2
[ndbd]
# Options for data node "A":
hostname=192.168.207.135 # Hostname or IP address
NodeId=3
[ndbd]
# Options for data node "B":
hostname=192.168.207.171 # Hostname or IP address
NodeId=4
[ndbd]
# Options for data node "C":
hostname=192.168.207.174 # Hostname or IP address
NodeId=5
[ndbd]
# Options for data node "D":
hostname=192.168.207.27 # Hostname or IP address
NodeId=6
[ndbd]
# Options for data node "E":
hostname=192.168.207.169 # Hostname or IP address
NodeId=7
[ndbd]
# Options for data node "F":
hostname=192.168.207.178 # Hostname or IP address
NodeId=8
[mysqld]
hostname=192.168.207.177
NodeId=10
[mysqld]
hostname=192.168.207.35
NodeId=11
[mysqld]
hostname=192.168.207.148
NodeId=12
My mysqld and ndb node config
######################
#API AND NDB CONFIG
######################
[mysqld]
ndbcluster
[mysql_cluster]
ndb-connectstring=192.168.207.133,192.168.207.45 # location of management server
Really hope someone can help. I've been at this for a month. We use data blobs quite extensively and I understand that this can cause these time-outs, but I'm most curious to find out why the 6-node restore sometimes succeeds and sometimes doesn't, and how I can go about ensuring the restore is successful every time.
I'm very open to try things and repost. I'm new to mysql cluster and have learned piles in the past few months, but am eager to learn more.
Thanks in advance,
GT
This is my my.cnf settings
[mysqld]
ndbcluster
#engine_condition_pushdown=0
optimizer_switch=engine_condition_pushdown=off
# IP address of the cluster management node
ndb-connectstring=127.0.0.1
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=127.0.0.1
[ndbd default]
NoOfReplicas= 2
MaxNoOfConcurrentOperations= 10000
DataMemory= 320M
IndexMemory= 96M
TimeBetweenWatchDogCheck= 30000
DataDir= /usr/local/mysql-cluster-gpl-7.2.2-osx10.6-x86_64/mysql-cluster
MaxNoOfOrderedIndexes= 512
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Section for the cluster management node
[NDB_MGMD]
# IP address of the management node (this system)
id=1
HostName=127.0.0.1
# Section for the storage nodes
[NDBD]
# IP address of the first storage node
id=2
HostName=127.0.0.1
DataDir= /usr/local/mysql-cluster-gpl-7.2.2-osx10.6-x86_64/mysql-cluster
[NDBD]
# IP address of the second storage node
id=3
HostName=127.0.0.1
DataDir= /usr/local/mysql-cluster-gpl-7.2.2-osx10.6-x86_64/mysql-cluster2
# one [MYSQLD] per storage node
[MYSQLD]
[MYSQLD]
I set data memory 320M and when I run script to fill table it stop with error 'table blabla is full'.
I try to run ndb_mgmd with -f option and with path of configuration file.
I copy this file in mysql-cluster folder with name config.ini but without result.
I got:
ndb_mgm> ALL REPORT MEMORYUSAGE
Node 2: Data usage is 100%(2560 32K pages of total 2560)
Node 2: Index usage is 74%(1741 8K pages of total 2336)
Node 3: Data usage is 100%(2560 32K pages of total 2560)
Node 3: Index usage is 74%(1741 8K pages of total 2336)
2560*32K=80M. 80M is like default mysql cluster settings.
i solve my problem. I need to divide this config on two config files one for mysql server and put it in data/my.cnf:
[mysqld]
ndbcluster
#engine_condition_pushdown=0
optimizer_switch=engine_condition_pushdown=off
# IP address of the cluster management node
ndb-connectstring=127.0.0.1
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=127.0.0.1
And second file for cluster bundle in /mysql_cluster/config.ini folder:
[ndbd default]
NoOfReplicas= 2
MaxNoOfConcurrentOperations= 10000
DataMemory= 320M
IndexMemory= 96M
TimeBetweenWatchDogCheck= 30000
DataDir= /usr/local/mysql-cluster-gpl-7.2.2-osx10.6-x86_64/mysql-cluster
MaxNoOfOrderedIndexes= 512
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Section for the cluster management node
[NDB_MGMD]
# IP address of the management node (this system)
id=1
HostName=127.0.0.1
# Section for the storage nodes
[NDBD]
# IP address of the first storage node
id=2
HostName=127.0.0.1
DataDir= /usr/local/mysql-cluster-gpl-7.2.2-osx10.6-x86_64/mysql-cluster
[NDBD]
# IP address of the second storage node
id=3
HostName=127.0.0.1
DataDir= /usr/local/mysql-cluster-gpl-7.2.2-osx10.6-x86_64/mysql-cluster2
# one [MYSQLD] per storage node
[MYSQLD]
[MYSQLD]
This is my config.ini file
options affecting ndbd processes on all data nodes:
[ndbd default]
NoOfReplicas=1 # Number of replicas
DataMemory=80M # How much memory to allocate for data storage
IndexMemory=18M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
TCP/IP options:
[tcp default]
portnumber=2202 # This the default; however, you can use any port that is free
# for all the hosts in the cluster
# Note: It is recommended that you do not specify the port
# number at all and allow the default value to be used instead
Management process options:
[ndb_mgmd]
hostname=192.168.1.10 # Hostname or IP address of management node
datadir=/var/lib/mysql-cluster # Directory for management node log files
Options for data node "A":
[ndbd]
# (one [ndbd] section per data node)
hostname=192.168.1.10 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
Options for data node "B":
[ndbd]
hostname=192.168.0.20 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
SQL node options:
[mysqld]
hostname=192.168.1.10 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
I want to specify second data node on 192.168.0.20 server.
What changes do i need to do on 192.168.0.20? It is a very low end machine so, I want don't want mysqld to run there.
can anyone provide any input on this.
If you use http://www.severalnines.com/config/index.php to create configurations files, and ask it to send it to you as an e-mail, you will get several scripts, including add_node.sh, which might be what you need.