I am having mysql 8.0 installation with replication between 2 servers in master-master configuration. While data is being replicated from server A to server B the replication is stalled from hours. Below is the snippet for show slave status \G on server B.
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 49808
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: waiting for handler commit
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
After executing show processlist; i am getting a process that is waiting from a long time , having a wait time of 50071 and increasing
system user | | NULL | Query | 50071 | waiting for handler commit
What steps shall i take to bring my replication on track?
Had the same problem. Adding:
innodb_flush_log_at_trx_commit = 2
Solved the problem for me on MySQL 8.0.13 installation.
Read here: https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
And see also: https://dba.stackexchange.com/questions/232365/mysql-8-0-updates-are-sometimes-stalling
Related
We are upgrading webservers and are trying to do the migration without downtime.
We're currently running MySQL 5.1.73 as a master and the new server (slave) is MariaDB 10.1.22
We have temporary tables in MySQL that get used to create statistics and then they are truncated.
The SQL that gets run is an INSERT..SELECT 0, (0 is for the primary column id that has auto increment)..
insert into tempStats select 0,tl.date,tl.sellerId...
I did a mysqldump from MySQL got the position, imported the data to MariaDB, started up the slave and then we keep getting the 'Duplicate entry' SQL errors.
Example:
MariaDB [(none)]> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event
Master_Host: SERVERIP_HERE
Master_User: st3slave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysqld-bin.001097
Read_Master_Log_Pos: 346910538
Relay_Log_File: main-dom0-relay-bin.000002
Relay_Log_Pos: 69379592
Relay_Master_Log_File: mysqld-bin.001093
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB: database_name_here
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1062
Last_Error: Error 'Duplicate entry '1923' for key 'PRIMARY'' on query. Default database: 'XYZ'. Query: 'insert into tempStats select 0,tl.date,tl.sellerId from tempLeadStatsSellersLeadsData tl left join tempLeadStatsSellers t on (t.date = tl.date AND t.sellerId = tl.sellerId AND t.type = tl.type AND t.subId = tl.subId) where isnull(t.date)'
Skip_Counter: 0
Exec_Master_Log_Pos: 410583345
Relay_Log_Space: 4300695570
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: Yes
Master_SSL_CA_File: /etc/mysql/sslcerts/ca-cert.pem
Master_SSL_CA_Path:
Master_SSL_Cert: /etc/mysql/sslcerts/new-live-server-cert.pem
Master_SSL_Cipher:
Master_SSL_Key: /etc/mysql/sslcerts/new-live-server-key.pem
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1062
Last_SQL_Error: Error 'Duplicate entry '1923' for key 'PRIMARY'' on query. Default database: 'XYZ'. Query: 'insert into tempStats select 0,tl.date,tl.sellerId from tempLeadStatsSellersLeadsData tl left join tempLeadStatsSellers t on (t.date = tl.date AND t.sellerId = tl.sellerId AND t.type = tl.type AND t.subId = tl.subId) where isnull(t.date)'
Replicate_Ignore_Server_Ids:
Master_Server_Id: 4
Master_SSL_Crl: /etc/mysql/sslcerts/ca-cert.pem
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: none
1 row in set (0.00 sec)
It's easy for us to do:
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; stop slave; start slave;
show slave status\G;
But the problem is these reports get rebuilt every 5 minutes. We thought it was the parallel mode and have set these options in Maria db cnf:
slave-parallel-threads=0
slave_parallel_threads=0
slave_parallel_mode=none
slave-parallel-mode=none
The SQL that it's failing on is a table that has the first column as the primary key that is on auto_increment and the query inserts "0" as the first column, so it seems like MariaDB is still doing some kind of parallel insertion here?
here's a copy of the server.cnf from the slave that's running MariaDB:
[mysqld]
slow_query_log = 1
slow_query_log_file = /var/log/mysql-slow.log
long_query_time = 2 # log_queries_not_using_indexes
slave which db to fetch from master
replicate-do-db=DB_NAME
end slave config
server_id=55
ssl
enable mysql replication over ssl
ssl
ssl-ca=/etc/mysql/sslcerts/ca-cert.pem
ssl-cert=/etc/mysql/sslcerts/new-live-server-cert.pem
ssl-key=/etc/mysql/sslcerts/new-live-server-key.pem
#master
#which db to dump to binlog
log-bin=mysql-bin
#lets log the updates we get from the master to binlog too
#so that the dev box can read these too
log-slave-updates
binlog-do-db=DB_NAME_HERE
#end master config
Anyone have any insight into why we keep getting these duplicate inserts errors?
I am creating the MySQL master-slave replication on CentOS 7 below are the configuration files details of both the server:
Master server's my.cnf
[mysqld]
server-id=1
log-bin=mysql-bin
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Slave Server's my.cnf
[mysqld]
server-id=2
replicate-wild-do-table=db1%.%
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Database details given below:
Master Server
create database db1;
create table db1.dbtb1(name varchar(100));
Slave Server
create database db1slave;
create table db1slave.dbtb1(name varchar(100));
AS you can see the Database name on both the sever are different for that I have use the below statement into the slave server for replication.
replicate-wild-do-table=db1%.%
But when I try to insert data into the Master Data base server I got the below error or slave server status:
Error 'Table 'db1.dbtb1' doesn't exist' on query. Default database: ''. Query: 'insert into db1.dbtb1 values ('Punu')'
Slave server Status:
Slave_IO_State: Waiting for master to send event
Master_Host: IP address
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000003
Read_Master_Log_Pos: 971
Relay_Log_File: mysqld-relay-bin.000002
Relay_Log_Pos: 283
Relay_Master_Log_File: mysql-bin.000003
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table: db1%.%
Replicate_Wild_Ignore_Table:
Last_Errno: 1146
Last_Error: Error 'Table 'db1.dbtb1' doesn't exist' on query. Default database: ''. Query: 'insert into db1.dbtb1 values ('Punu')'
Skip_Counter: 0
Exec_Master_Log_Pos: 760
Relay_Log_Space: 668
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1146
Last_SQL_Error: Error 'Table 'db1.dbtb1' doesn't exist' on query. Default database: ''. Query: 'insert into db1.dbtb1 values ('Punu')'
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_UUID: 2fc2ef76-c87a-11e6-ae22-000d3aa2da57
Master_Info_File: /var/lib/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 161227 18:40:36
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
If I will take the same database name it will work fine. But as per my requirement I can't put the database name same on both the server.
Simply I want that when the table "dbtb1" updated in Master server database then it will be replicated to the slave database table.
I have also try the below options in the slave my.cnf file:
1
replicate-wild-do-table=db1%.dbtb1
2
replicate-rewrite-db="db1->db1slave"
Above option will work fine if I insert record but it will not reflect when I delete or update record.
3
replicate-wild-do-table=%.dbtb1
your option replicate-wild-do-table=db1%.% will replicate all table in all schemas starts with db1% ans also replicate all tables start with %. so it will replicate all tables. thats a wrong config.
use the replicate-rewrite-db option to config this. read this below.
https://mariadb.com/resources/blog/multisource-replication-how-resolve-schema-name-conflicts
Using Percona MySQL 5.6 with sql_slave_parallel_workers=5 on Debian 8. Sometimes GTID replication breaks and I don't know why. I thought that the GTIDs are executed in a consecutive order, but when looking at status
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: d22.local
Master_User: xyz
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.039232
Read_Master_Log_Pos: 219044
Relay_Log_File: mysqld-relay-bin.072392
Relay_Log_Pos: 90640
Relay_Master_Log_File: mysql-bin.036196
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB: xyz_etl
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1032
Last_Error: Could not execute Update_rows event on table xyz.sessions; Can't find record in 'sessions', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.036196, end_log_pos 78709552
Skip_Counter: 0
Exec_Master_Log_Pos: 78708927
Relay_Log_Space: 1337994488
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Update_rows event on table xyz.sessions; Can't find record in 'sessions', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.036196, end_log_pos 78709552
Replicate_Ignore_Server_Ids:
Master_Server_Id: 22
Master_UUID: 0e7b97a8-a689-11e5-8b79-901b0e8b0f53
Master_Info_File: /var/lib/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 161219 20:32:20
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 0e7b97a8-a689-11e5-8b79-901b0e8b0f53:60397-45157441
Executed_Gtid_Set: 0e7b97a8-a689-11e5-8b79-901b0e8b0f53:1-42679868:42679870-42679876:42679878-42679879:42679881-42679890:42679892-42679908:42679910:42679913:42679916-42679917:42679919-42679927:42679929-42679932:42679934:42679936:42679938-42679939:42679944:42679946-42679950:42679952-42679955:42679957-42679964:42679966:42679969-42679970:42679972:42679974-42679977:42679979-42679980:42679984-42679986:42679988-42679990:42679994-42679996:42679998:42680000-42680001:42680003-42680006:42680009-42680011:42680013-42680018:42680021:42680024:42680026:42680030:42680032:42680035:42680038,
aea3618e-bacf-11e6-9506-b8ca3a67f830:1-10937274
Auto_Position: 1
1 row in set (0.00 sec)
I'm a bit confused. sql_slave_parallel_workers is set to 0 now. But the error claimed above is GTID 42679909 instead of 42679868 as expected. What's the reason for this. And what are the correct steps to solve a broken replication like above?
What I don't understand is, that the transaction with GTID 42679869 can be executed without problems, theoretically. But doing a STOP SLAVE; START SLAVE; does not process them?!
To answer it and help others, here the steps I've done:
setting slave_parallel_workers=0
one have to pay attention to field Executed_Gtid_Set only and handle all gaps in GTID list one after another with STOP SLAVE; SET GTID_NEXT="[...]"; BEGIN; COMMIT; SET GTID_NEXT="AUTOMATIC"; START SLAVE;
when point is reached, that replication will continue automatically without error set slave_parallel_workers to previous value
I am running this query in my slave machine.
Query:
select ID from audit where app='accounts'
explain output
+----+-------------+-----------------+------+---------------+------+---------+------+-----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------+------+---------------+------+---------+------+-----------+-------------+
| 1 | SIMPLE | IAMAccountAudit | ALL | NULL | NULL | NULL | NULL | 155658522 | Using where |
+----+-------------+-----------------+------+---------------+------+---------+------+-----------+-------------+
After Execution , my slave machine was running behind the master.
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 182.31.251.94
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: bin.001487
Read_Master_Log_Pos: 2967065
Relay_Log_File: 172-relay-bin.004312
Relay_Log_Pos: 43303861
Relay_Master_Log_File: bin.001486
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 43303721
Relay_Log_Space: 55397036
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 365
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1302078
It's keep increasing .
How can I avoid the mysql backlog? .
It would help a bit if I had a table schema to work with. However from what I can see your query is not using a index and sequentially reading through the table from the first record to the last record one by one.
As the query has to examine roughly 155,658,522 or 155 million rows it is going to cause a resource drain on the server. Since the query has no appropriate index on the column you are querying(key= null in explain output) the rows are getting a read lock issued on them one by one.
While this read is happening it is possible for MySQL to block the replication activity i.e. block updates to the rows being read.MySQL will then queue these updates as they cannot complete. This problem is due to ACID compliancy of the server and not threading etc. Since I dont have table information such as engine this is all a educated guess at best.
Suggestions:
Index the app column on the table allowing you to do the select faster. With a index on the app column MySQL will do a b-tree search on the table instead of a sequential search. You query will run faster issuing less locks thus it will be faster and more light weight. The downside is that it will slightly delay updates and inserts on this table as there is an additional index to maintain.
Unless you need to the nano-second real time replication leave the system as is. Keep in mind that replication falling behind is not a problem if there is no requirements for real time data. Replication failing is much worse.
Hope this helps a bit.
I have a primary MySQL server and am trying set up a replication for InnoDB database.
I perform a Master and Slave configuration, create a dump of Master station DB, restore it on Slave station and run replication. Everything looks fine but no updates are reflected on Slave. The Slave is running, the Read_Master_Log_Pos on Master and Slave are identical. I even look up the mysql-relay-bin.00000X file and there are updates from the Master. But no changes when I run "SELECT * FROM db.table;".
If I change BD engine to MyISAM everything works fine.
I also tried to create an empty database (InnoDB engine), dump it and restore from it on the Slave, start replication and then create tables on Master and fill them. In this case replication is doing well. But I also need a way to replicate already created and filled database.
What should I do?
I have:
my.cnf on Master:
[mysqld]
bind-address=0.0.0.0
port=3306
replicate-do-db=gcm_data
log-bin=/var/lib/mysql/mysql-bin
server-id=8217
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
my.cnf on Slave:
[mysqld]
bind-address=0.0.0.0
port=3306
replicate-do-db=gcm_data
relay-log-index=/var/lib/mysql/mysql-relay-bin.index
relay-log=/var/lib/mysql/mysql-relay-bin
server-id=50890
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
SHOW MASTER STATUS\G
File: mysql-bin.000001
Position: 657
Binlog_Do_DB:
Binlog_Ignore_DB:
SHOW SLAVE STATUS\G
Slave_IO_State: Waiting for master to send event
Master_Host: 10.0.13.52
Master_User: slave1
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 657
Relay_Log_File: mysql-relay-bin.000004
Relay_Log_Pos: 251
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB: gcm_data
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 2
Exec_Master_Log_Pos: 657
Relay_Log_Space: 551
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
SHOW PROCESSLIST on Master:
| Id | User | Host | db | Command | Time | State | Info |
| 16 | slave1 | 10.0.13.140:54683 | NULL | Binlog Dump | 2062 | Has sent all binlog to slave; waiting for binlog to be updated | NULL |
SHOW PROCESSLIST on Slave:
| Id | User | Host | db | Command | Time | State | Info |
| 14 | system user | | NULL | Connect | 2203 | Waiting for master to send event | NULL |
| 15 | system user | | NULL | Connect | 2203 | Has read all relay log; waiting for the slave I/O thread to update it | NULL |
If it is somehow important I'm running both Master and Slave station from virtual CentOS 6.0, MySQL 5.1.61.
UPDATE
I implemented a little behavior test: I created am empty database on Master, run a replication script (everything went well - Master and Slave were running). Then I created an InnoDB table
CREATE TABLE tbl (id INT, data VARCHAR(10)) engine=InnoDB;
The table appeared on Slave. Then I filled table with data
INSERT tbl values (1, "1");
The data was updated on Slave.
Then I stopped Master (service mysqld stop) and Slave IO also stopped
SHOW SLAVE STATUS - Slave_IO_Running: No, Slave_SQL_Running: Yes
Then I started MySQL on Master and Slave IO also started in some time. But since updates are not available on Slave, hence they are present in mysql-relay-bin.00000X on Slave.
Well.. I somehow solved the problem. It seems that the problem is exactly in running both Master and Slave from virtual machines. I run the same replication script on real machines and everything is working just fine.
So now the question is why isn't it working on VM properly.
UPDATE
Some time ago same problem raised on the real machines.
And eventually I found the source of this problem:
I was testing my replication using INSERT/SELECT commands like this:
INSERT db.table (ID, VALUE) values (ID, 2);
on Master and checking:
SELECT * FROM db.table;
on Slave. And id didn't show me the updates.
But then I tried:
USE db;
INSERT table (ID, VALUE) values (ID, 2);
And update was available on Slave station.
Check status by command show slave status\G
stop slave;
reset slave all;
At Slave Server show slave status\G;
stop slave;
reset slave all;
drop database dbname; (make sure to delete slave db)
create database slavedb;
restore backup of master db to slave db
check head count by command "head -25" to get log position from your newly restored backup file.
add binary file name
CHANGE MASTER TO MASTER_HOST = 'Master IP', MASTER_USER = 'user', MASTER_PASSWORD = 'password', MASTER_LOG_FILE='mysql-masterdb-bin.002428', MASTER_LOG_POS=57508035;
start slave;