Is there a way to see when the last byte of data was copied over from the master to the slave? Currently to check how 'current' the data is I'm doing some pretty crude such as:
select max(last_updated) from one-my-my-tables
But it doesn't work too well. Is there a more formal way to do this?
From a privileged MySQL account (e.g. root) use:
mysql> show slave status;
The field:
Seconds_Behind_Master: 0
tells you how far out of date the slave is.
Unfortunately, there's no way to get the information from SHOW SLAVE STATUS without using that statement. I searched for this recently and learned that there are replication info tables in the PERFORMANCE_SCHEMA, however none of them contain the Seconds_Behind_Master.
Strictly speaking, the Seconds_Behind_Master doesn't tell you what you asked, anyway. You asked "when the last byte of data was copied over from the master to the replica?" Seconds_Behind_Master tells you the difference between the system time on the replica vs. the timestamp of the last executed event from the relay log.
Or if the replica has executed all downloaded events, it reports 0.
But suppose the replica has lost contact with its master, and there are more logs sitting around on the master waiting to be downloaded? The replica doesn't report this, because it has no idea there are more logs.
A more accurate way of measuring replication lag is to use the pt-heartbeat script, which is included in the free Percona Toolkit.
You execute the script on the master, and it inserts a timestamp to a table once per second, like a heartbeat.
Then on the replica you can query the timestamp and compare it to the system time.
If the replica is caught up, the difference will be zero.
If the replica has downloaded all the logs, but executing events is lagging, the timestamp difference will show this.
If the replica has lost contact with its master and hasn't downloaded all the logs, but we know the heartbeat timestamp should update once per second, then you can still get an accurate measure of the replication lag.
Related
We have a one master and two VIP slave database servers. We changed data type of column from VARCHAR(255) to TEXT on master.
The application is currently configured to use master only for writing operations and configured slaves for reading operation.
After changing the data type on master server using ALTER TABLE command the slave server becomes unresponsive.
We are using Mariadb 10.0
[PROCESSES INFORMATION]
Id User Host Db Command Time(sec) State Info
-----------------------------------------------------------------------
203739 repl slave1 Binlog Dump 75,143,121 Master has sent all binlog to slave; waiting for binlog to be updated
203740 repl slave2 Binlog Dump 75,143,121 Master has sent all binlog to slave; waiting for binlog to be updated
The slave instance becomes very slow due to slow queries.
number of sessions: 1590
thread_pool_max_thread=500
Current value =648
After performing ALTER TABLE on Master server, it was replicating to slave server and in the same time number of sessions were get increased rapidly on slave server.
I think slaves becomes unresponsive because of slow queries.
But I don't know why this queries became so slow and slaves got unresponsive.
The DBA's saying that after executing ANALYZE TABLE command, the issue has been solved.
But I don't understand why this happened because ANALYZE TABLE only update the statistic information.
It would be helpful if anyone comment on this why it happened?
How to avoid such issues in future.
There is one minor case where TEXT is slower than VARCHAR. When a SELECT needs to build a temporary table (often for sorting due to GROUP BY or ORDER BY), it first tries to build a MEMORY table. But, TEXT and BLOB prevent it from using such, so it uses MyISAM instead. This is slower (but gets the job done).
I say this is a "minor case" because users rarely identify it with phrases like "very slow" and "becomes unresponsive". I would guess that a SELECT might run twice as slow.
Also, the ANALYZE TABLE discussion does not hold water. Again it may be coincidence, not causation.
So, the change to TEXT may be a 'red herring'. Instead, let's discover what is being slow by using the slowlog. See this for what I like to work from.
With a replicating mysql master to master database with innodb engine, if one transaction were to initiate on database A will that row lock for database B until the transaction has been committed?
The master getting the first transaction is completely separate from the second master and they communicate through a binary log.
https://dev.mysql.com/doc/refman/5.7/en/replication-formats.html
In the case of something requiring a transaction, then the actual statements are not written to the log until the transaction is complete.
https://dev.mysql.com/doc/refman/5.7/en/replication-features-transactions.html
So the second master should be completely unhindered, since it won't actually know anything about the request until the first master is done processing it.
(Standard caveats though of it may depend on what type of replication SBR/RBR/mix and the actual transactions.)
I have a number mysql servers running version 5.1.63 and whilst running some queries against the slave earlier this week, I noticed some data on the slave that should have been removed using an update statement on the master.
My initial thoughts were:
someone on the team was updating the slave, which I have since disproved
that the column being updated had changed
So, I investigated by running a mysql show status "table" query. This was run against a test database on each of the servers to see what the data length was, in a lot of cases it was showing me the data length differed between servers, but on an eyeball look at the data I could see the data was the same, so I couldn't use this method to see if there were any differences as it appears to be prone to error.
Next I ran a simple (across all dbs) row count for each table to confirm the row count was the same - it was.
I then started looking in the bin logs for replication. I could see the update statements that should have run clearly visible in the logs, but the update never ran.
What I need to know is:
is replication broken? I'm assuming it is
if I create new slave servers, will I encounter the same issue?
how do I find out the extent of the issue on my servers?
Any help is appreciated.
If you are using statement based replication then it is easily possible to end up with different results on master and slave due to badly constructed INSERT statements.
INSERT SELECT without ORDER BY, or where the ORDER BY can leave non deterministic results will cause the slaves to diverge from master.
From the MySQL site http://dev.mysql.com/doc/refman/5.1/en/insert-select.html
The order in which rows are returned by a SELECT statement with no
ORDER BY clause is not determined. This means that, when using
replication, there is no guarantee that such a SELECT returns rows in
the same order on the master and the slave; this can lead to
inconsistencies between them. To prevent this from occurring, you
should always write INSERT ... SELECT statements that are to be
replicated as INSERT ... SELECT ... ORDER BY column. The choice of
column does not matter as long as the same order for returning the
rows is enforced on both the master and the slave. See also Section
16.4.1.15, “Replication and LIMIT”.
If this has happened then your replicas have diverged and the only safe way to bring them back in line is to rebuild them from a recent backup of the master DB. The worst part of this is the error may never cause the replication to fail, yet the results are inconsistent. Normally replication fails when an UPDATE or DELETE statement affects a different number of rows than on master, this is confusing as it was not the UPDATE that actually caused the error and the only way I know to fix the issue is to inspect every INSERT query in the code base!
Status details are from information_schema which collates data from databases statistics for Mysql instance and it never remained the same at every execution. It can be considered as just a rough estimation of data sizes in bytes but never an exact value as for index and data length. It can be used for estimations but not for cross check. For replication you may check the slave io and sql against the master is running or not. And relay-info you might see the corresponding log details from master and that of slave.
Of,course (1) way of doing is count(*) of tables EOD ensures the data in tables on master and slave are consistent or not. But to be accurate either (2) take random value fields and cross check with master and slave. Also if you aren't satisfied with it, (3) you may take them into outfile and take diff or checksum. I prefer (1) and (2). If (1) is not possible (2) still convinces me. ;)
There is a tool to verify replication named pt-table-checksum
I have a Master to Master replication set up between two MySQL 5.0
My requirement is that before starting my application I have to ensure that the database are identical I would like to confirm that the "seconds_behind_master" of the "show slave status" command being at 0 seconds is enough to consider the 2 databases being synchronized ?
http://dev.mysql.com/doc/refman/5.0/en/show-slave-status.html
Thank you
No, that is not sufficient to guarantee that the databases are identical, just that the slave has run all statements from the master's binary log. You could update or delete large chunks of data on the slave database and still have seconds behind master = 0, but the slave would certainly not be identical to the master.
You should use a tool like Percona's pt-table-checksum if you really want to verify that the databases are identical.
It's not really a guarantee, as master-to-master replication does not guarantee atomicity across servers.
Also, a single read of 0 could be meaningless, but a consistent read of 0 over time would be a good indication that the servers are synced, if you have no errors reported.
So, I say, read it every second for 10 seconds, and if they're all 0, with no errors, you're probably good enough, short of comparing checksums, as Ike suggested.
I have 2 servers set up using MySQL. It's using a standard replication setup, with one slave, no circular replication.
Is there a way to programmatically tell how far behind the slave is in reading the data from the binary log?
If I run the statement:
SHOW MASTER STATUS;
On the master, and run
SHOW SLAVE STATUS;
on the slave, I can compare the Position column from master status, and the Read_Master_Log_Pos column from slave status to determine how far behind the slave is.
However, this only works if the slave is reading from the same file the master is writing to. So if the slave is still reading a previous log file, because it is running behind, I can't figure out how to determine how much data is left until it catches up to the current position that the master is at. A solution using only SQL would be optimal, but I'm open to other solutions. Hopefully not one that requires reading the directory containing the log files.
I like to use the 'Seconds_behind_master' field from SHOW SLAVE STATUS in order to determine if the MASTER-SLAVE servers are caught up. As a secondary guard I also to a COUNT(*) query on a specific table (ie one that gets updated frequently) on both servers and then compare the record counts.