I have a Master to Master replication set up between two MySQL 5.0
My requirement is that before starting my application I have to ensure that the database are identical I would like to confirm that the "seconds_behind_master" of the "show slave status" command being at 0 seconds is enough to consider the 2 databases being synchronized ?
http://dev.mysql.com/doc/refman/5.0/en/show-slave-status.html
Thank you
No, that is not sufficient to guarantee that the databases are identical, just that the slave has run all statements from the master's binary log. You could update or delete large chunks of data on the slave database and still have seconds behind master = 0, but the slave would certainly not be identical to the master.
You should use a tool like Percona's pt-table-checksum if you really want to verify that the databases are identical.
It's not really a guarantee, as master-to-master replication does not guarantee atomicity across servers.
Also, a single read of 0 could be meaningless, but a consistent read of 0 over time would be a good indication that the servers are synced, if you have no errors reported.
So, I say, read it every second for 10 seconds, and if they're all 0, with no errors, you're probably good enough, short of comparing checksums, as Ike suggested.
Related
Is there a way to see when the last byte of data was copied over from the master to the slave? Currently to check how 'current' the data is I'm doing some pretty crude such as:
select max(last_updated) from one-my-my-tables
But it doesn't work too well. Is there a more formal way to do this?
From a privileged MySQL account (e.g. root) use:
mysql> show slave status;
The field:
Seconds_Behind_Master: 0
tells you how far out of date the slave is.
Unfortunately, there's no way to get the information from SHOW SLAVE STATUS without using that statement. I searched for this recently and learned that there are replication info tables in the PERFORMANCE_SCHEMA, however none of them contain the Seconds_Behind_Master.
Strictly speaking, the Seconds_Behind_Master doesn't tell you what you asked, anyway. You asked "when the last byte of data was copied over from the master to the replica?" Seconds_Behind_Master tells you the difference between the system time on the replica vs. the timestamp of the last executed event from the relay log.
Or if the replica has executed all downloaded events, it reports 0.
But suppose the replica has lost contact with its master, and there are more logs sitting around on the master waiting to be downloaded? The replica doesn't report this, because it has no idea there are more logs.
A more accurate way of measuring replication lag is to use the pt-heartbeat script, which is included in the free Percona Toolkit.
You execute the script on the master, and it inserts a timestamp to a table once per second, like a heartbeat.
Then on the replica you can query the timestamp and compare it to the system time.
If the replica is caught up, the difference will be zero.
If the replica has downloaded all the logs, but executing events is lagging, the timestamp difference will show this.
If the replica has lost contact with its master and hasn't downloaded all the logs, but we know the heartbeat timestamp should update once per second, then you can still get an accurate measure of the replication lag.
we have a MySQL with a replica (5.7 with row based replication).
Now, the master performs at peak about 3000 inserts per second, and the replica seems to read that just fine. However, sometimes we execute long-time select queries (that ran from 10 to 20 seconds). And during those queries the replication lag becomes very huge.
What I do not understand is how the usual mysql threads that execute selects (without locking any tables) can cause the replication thread to slow down (i.e. it performs about 2.5K inserts instead of 3K like master)? What would I need to tune exactly?
Now I checked the slave status and it's not about the IO thread - this one manages to read events from the master just fine. It's SQL slave thread, that somehow does not manage to catch up. The isolation level is Read Committed, so the select queries potentially could lock some records and make the slave thread wait. But I'm not sure about that.
UPDATED. I have checked again - it turns out that even a single heavy query (that scans the entire table for example) on the slave produces the lag. It seems like slave sql thread is blocked, but I do not understand why?
UPDATED 2. I finally found the solution. First I increased number of slave_parallel_workers to 4 and set slave_parallel_type to LOGICAL_CLOCK. However, and this is important, that gave me no improvement at all, since the transactions were dependent. But, after I increased on master binlog_group_commit_sync_delay to 10000 (that is, 10 milliseconds), the lag disappeared.
There Might be many reasons why replication lag in mysql slave database.
But as you mentioned
It's SQL slave thread, that somehow does not manage to catch up.
Assuming that IO works fine, Percona says (emphasis mine):
[...] when the slave SQL_THREAD is the source of replication delays it is probably because of queries coming from the replication stream are taking too long to execute on the slave. This is sometimes because of different hardware between master/slave, different schema indexes, workload. Moreover, the slave OLTP workload sometimes causes replication delays because of locking. For instance, if a long-running read against a MyISAM table blocks the SQL thread, or any transaction against an InnoDB table creates an IX lock and blocks DDL in the SQL thread. Also, take into account that slave is single threaded prior to MySQL 5.6, which would be another reason for delays on the slave SQL_THREAD.
We have a one master and two VIP slave database servers. We changed data type of column from VARCHAR(255) to TEXT on master.
The application is currently configured to use master only for writing operations and configured slaves for reading operation.
After changing the data type on master server using ALTER TABLE command the slave server becomes unresponsive.
We are using Mariadb 10.0
[PROCESSES INFORMATION]
Id User Host Db Command Time(sec) State Info
-----------------------------------------------------------------------
203739 repl slave1 Binlog Dump 75,143,121 Master has sent all binlog to slave; waiting for binlog to be updated
203740 repl slave2 Binlog Dump 75,143,121 Master has sent all binlog to slave; waiting for binlog to be updated
The slave instance becomes very slow due to slow queries.
number of sessions: 1590
thread_pool_max_thread=500
Current value =648
After performing ALTER TABLE on Master server, it was replicating to slave server and in the same time number of sessions were get increased rapidly on slave server.
I think slaves becomes unresponsive because of slow queries.
But I don't know why this queries became so slow and slaves got unresponsive.
The DBA's saying that after executing ANALYZE TABLE command, the issue has been solved.
But I don't understand why this happened because ANALYZE TABLE only update the statistic information.
It would be helpful if anyone comment on this why it happened?
How to avoid such issues in future.
There is one minor case where TEXT is slower than VARCHAR. When a SELECT needs to build a temporary table (often for sorting due to GROUP BY or ORDER BY), it first tries to build a MEMORY table. But, TEXT and BLOB prevent it from using such, so it uses MyISAM instead. This is slower (but gets the job done).
I say this is a "minor case" because users rarely identify it with phrases like "very slow" and "becomes unresponsive". I would guess that a SELECT might run twice as slow.
Also, the ANALYZE TABLE discussion does not hold water. Again it may be coincidence, not causation.
So, the change to TEXT may be a 'red herring'. Instead, let's discover what is being slow by using the slowlog. See this for what I like to work from.
I have an account in amazone. my primary MySql is a large instance. and i have other 2 small instances as replication servers. The hierarchy is Primary -> slave 1-> slave 2 . The problem is that some times the slave 1 and slave 2 showing high CPU utilization. We couldn't find out the exact reason. Slave 1 is act as a slave of primary and at the same time it act as Master of Slave 2. We searched a lot but we are still stucked as blind.
Thanks in advance for all helps.
As a general rule, when using MySQL native, asynchronous replication -- which is what RDS uses -- the replica servers need be as large as the master or sometimes larger. The replicas receive "replication events" from the master, which may contain the actual queries that modified data -- insert, update, delete but not select -- this is "statement based replication" -- or may receive binary images of the rows added removed or changes on the master -- which is "row based replication." By default, the master server decides, on a query-by-query basis, which format to use to send the replication events ("mixed").
In all cases SELECT statements are not sent to the replicas (except of course for INSERT ... SELECT), but they do need sufficient capacity to handled both the incoming changes from the master, as well as the SELECT queries that are run directly against the replica by your application.
In RDS for MySQL 5.6 and above,you can set the binlog_format to ROW to force the master to always use row-based replication. This might improve your performance, and it might not -- it depends on the workload. You cannot force RDS to use only "statement" mode, nor should you want to.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html
In general, however, any time the replica is a machine with fewer resources than the master, the chance of replication lags increases. The lag of the replicas can be monitored in Cloudwatch.
Lets say I have a datbase of Cars. I have Makes and Models (FK to Makes). I plan on having users track their cars. each Car has a FK to Model. Now, I have a lot of users, and I want to split up my database to distribute load. The Makes and Models tables don't change so much, but they need to be shared across shards. My thought is to use MySQL replication from a master DB of makes and models to each slave database. My question is: Can I safely write to the slave databases assuming I don't write to those tables on the master?
And while on the subject, is there anyway to guarantee one slave database has the latest data? For example, someone just added the 'Taurus' make, and then wants to add their car. Can I ensure that the slave database they are using has the latest master data?
Yes, in general you can safely write to a table on the slaves that is not being written on the master. If you do things like insert auto_increment rows on the slaves and on the master, independently, you will of course have problems. You should configure that table to be excluded from replication entirely, really.
For checking whether you have the latest data, SHOW SLAVE STATUS includes a field Seconds_Behind_Master that tells you whether the slave is up to date. Obviously you want it to be zero. To be certain that inserted and replicated data is present, of course, you need to wait a second and then see that Seconds_Behind_Master is zero.
This was a good solution I gleaned while searching
I included the main point as avilable here:
http://erlycoder.com/43/mysql-master-slave-and-master-master-replication-step-by-step-configuration-instructions-
MySQL master-master replication and autoincrement indexes
If you are using master-slave replication, than most likely you will design your application the way to write to master and read from slave or several slaves. But when you are using master-master replication you are going to read and write to any of master servers. So, in this case the problem with autoincremental indexes will raise. When both servers will have to add a record (different one each server simultaneously) to the same table. Each one will assign them the same index and will try to replicate to the salve, this will create a collision. Simple trick will allow to avoid such collisions on MySQL server.
On the Master 1/Slave 2 add to /etc/my.cnf:
auto_increment_increment= 2
auto_increment_offset = 1
On the Master 2/Slave 1 add to /etc/my.cnf:
auto_increment_increment= 2
auto_increment_offset = 2