I'm mysql newbie.
when it comes to fail-over, which slave should be promoted to the new master?
For example, A is master, B and C are slaves, and A does async replication to B and C.
At some point of time, B receives more data from A than C, A crashes.
If we promote C to new master, and changes B's master to C, then what happens to B? It truncates its data to match C?
Obviously, B is the best new master candidate, but my question is, how to determine this fact?
From the MySQL documentation, there two ways to set up a master-slave architecture. The traditional way, using the log files to replicate transactions and the new version (5.6+) using GTIDs (global transaction identifiers).
If you choose to use GTIDs to make the failover handling you will use the mysqlfailover utility. The utility handles fails of master in one of three ways defined by the database administrator:
auto (default): A search is made in a list of prefered slaves to become master, if none are available another slave is chosen. The chosen slave first becomes the slave to all others slaves and has all the changes from the other slaves copied to it, this way the new master will be the most up to date version possible.
elect: the same as above, except that if no slaves from the list are available it returns an error and finishes (no failover)
fail: No failover happens mysqlfailover will just monitor the database and return an error if a fail happens.
The traditional way requires that you implement your own script to database management and is better explained here.
The Relay_Master_Log_File and Exec_Master_Log_Pos in SHOW SLAVE STATUS is used to determine the best slave as new master: the bigger values win.
Without GTID, I think we must first sync other slaves with the best slave we chose. The obvious sync source is the relay logs. On each slave, determine the differences of relay log from the best slave, download those files and replay the SQL statements. Once all slaves catch up, the slaves could CHANGE MASTER TO the best slave. The MASTER_LOG_FILE and MASTER_LOG_POS would be the tail of the last binlog on the best slave.
With GTID, it's very simple: just CHANGE MASTER TO with MASTER_AUTO_POSITION=1.
Related
I've set up a slave replication of MySQL database. And for development requirements, I want to write sth into the slave database, but it would cause the broken of replication.
Since the database is huge, I don't want to restore the slave database from MySQL dump file every time after I finished some development work.
My requirement:
All the changes in the slave database can be reverted by a simple command.
The replication keeps working.
One method is to use LVM filesystem snapshots. Before you begin testing:
Stop replication.
Take an LVM snapshot.
Do your tests. Replication is still off, but data is up to date
After you finish testing:
Stop mysqld.
Restore the snapshot. This reverts all files to the state they were at the moment you created the LVM snapshot above.
Start mysqld and start replication. It will need to catch up and apply all changes since you stopped replication before your testing. This will take a little while, depending on how many changes happened on your master database.
See https://www.tecmint.com/take-snapshot-of-logical-volume-and-restore-in-lvm/ for a nice tutorial on using LVM snapshots.
This method only works if your development database instance is on Linux.
Insert the new records using a primary key that is not expected to be used by the master database (e.g. add a sufficiently large offset like 2^10 or negative numbers if allowed...).
In this way, the insertions coming from the master won't clash.
I have remote host (centos6) with mariadb (10.0.17-MariaDB-log - MariaDB Server) as master:
server-id = 1
log-bin=mysql-bin
binlog_do_db = mydatabase
and local (win8.1) with mariadb (10.0.16-MariaDB-log - mariadb.org binary distribution) as a slave:
server-id = 2
As initial procedure I've dumped database on remote host, imported it on local host, then executed SHOW MASTER STATUS, get filename and offset and run:
CHANGE MASTER TO MASTER_HOST='$host', MASTER_USER='$user', MASTER_PORT = $port, MASTER_PASSWORD='$pass', MASTER_LOG_FILE='$fname', MASTER_LOG_POS=$pos
STOP SLAVE
START SLAVE
Replication starts. Everything I do with table on master is reflected to slave.
But if slave goes down, after it is up no changes (made on master) while slave was offline reflected to slave! So it looks like my slave should always be online, but it's a laptop!
However after slave is up realtime replication still works - it doesn't remember all changes from offline, but if I change database on master when slave is online and started all changes are perfectly reflected to slave. Of course I know that replication is statement based, so I get not data diff but instructions. But I thought master remembers what it sent and what did not. So on next operation it just sends all non-delivered changes. Am I wrong?
My replication scenario: master server interacts with clients (mobile devices) and they change the database. From time to time I launch my laptop, start replication, get updated database and do some heavy analysis (it's too hard for my 2-core cheap server).
Maybe there is a better method? Is there a way to get "offline changes" like in ICQ messenger? :)
For now I can see only one solution - full db dump, but it is inconvenient, takes too much time and loads master heavily.
While the Slave is not connected to the Master, the Master is writing to its binlog(s). The Slave has remembered where it left off in reading from those binlogs. When the Slave reconnects, it picks up "where it left off", copies the changes from the Master's binlog(s) to the Slaves relay-log(s) and performs them. This "catchup" process will take a little time, how long depends on a lot of factors.
Do SHOW SLAVE STATUS; on the Slave to verify that it is connected and running ("Yes").
Normally, the version of the Slave should be no older than that of the Master. (I doubt if there is any issue between 10.0.17 vs 16. Nothing in the changelog for 10.0.17 jumps out at me.)
Are you using "parallel replication"? See bug fixed in 10.0.18.
I'm running MySQL as the database on Ubuntu instances. I'm using MySQL Master-Slave replication where master's changes will be written to slave and slave's changes will not be reflected on the master. That's fine. I'm using a HAProxy load balancer to front the MySQL instances where all the requests will be sent to master MySQL instance. If the master MySQL instance is down slave MySQL instance will act as master and HAProxy will send all the requests to salve. Active-Passive scenario.
HAProxy - 192.168.A.ABC
MySQL Master - 192.168.A.ABD
MySQL Slave - 192.168.A.ABE
Let's assume that the MySQL master(192.168.A.ABD) is down. Now all the requests will be sent to MySQL slave(192.168.A.ABE) by HAProxy where now he acts as the master MySQL server for the time being.
My problems are
What happens when original master MySQL instance(192.168.A.ABD) is up?
Will changes written to new MySQL master (192.168.A.ABE) be replicated to original master(192.168.A.ABD) again?
How should I address this scenario?
First of all I should say that I have never used HA Proxy so con't comment on that directly.
However, in your current setup the Master (ABD) will be out of sync, and wont catch up. You will have to rebuild that using mysqlDump or similar tool.
What you would need is a Master < > Master setup (as opposed to Master > Slave), which enables you to write to either database and have it reflected in the other. This isn't quite as straight forward as it sounds though.
Assuming you already have your master > slave setup, and they are in sync
On the Master (ABD) you want to add:
auto_increment_increment=2
auto_increment_offset=1
log-slave-updates
On the Slave (ABE) add:
auto_increment_increment=2
auto_increment_offset=2
log-slave-updates
to your my.cnf files. Restart the Database. This will help to prevent Duplicate Key Errors. (n.b. that log-slave-updates isn't strictly required but makes it easier to add another slave in future)
Next you want to tell the Master (ABD) to replicate from the Slave (ABE).
Depending on what version of MySQL and if you are using GTID etc. the exact process differs slightly. But basically you are going to issue a CHANGE MASTER statement on the Master so it replicates from the slave.
And away you go. You probably want to avoid writing to both at the same time as that opens up a whole other kettle of fish. But if the Master goes down, you can switch your writes to the slave, and when the master comes back up, it will simply start replicating the missing data.
I am considering you scenario
Master - 192.168.A.ABD
Slave - 192.168.A.ABE
You cannot directly add the master in system. To Add master in system you need to perform below steps:
1) When master is up you can add this as a slave. So now this happens
Master - 192.168.A.ABE
Slave - 192.168.A.ABD
2) Then Now U can put master Down. Means You can put 192.168.A.ABD Down
3) Then Again Add this as slave. So After this You will get below scenarion
Master - 192.168.A.ABD
Slave - 192.168.A.ABE
You can refer this link
https://dev.mysql.com/doc/refman/5.5/en/replication-solutions-switch.html
I have a redis master setup with 2 slaves and a sentinal on each.
If the master goes down for say 2 seconds (+sdown) and comes back up (-sdown), it reads the last snapshot, and the slaves resync with the master.
The problem with this is that there may have been many writes between the last save and the +sdown. It seems like if the master goes from +sdown to -sdown and never +odown (where a failover is initiated), it should be able to sync FROM a slave. My reasoning is that the replication stream is continuous and the slaves most likely have a more accurate reflection of the masters state when +sdown happened.
Is there some config that I can do this? Am I forced to rely on the AOF or snapshots?
(Edit: adding sentinel tag)
You cannot do a partial failover, either you do it or you don't in terms of promoting the slave to master.
From Redis Sentinel:
Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the other additional slaves are reconfigured to use the new master, and the applications using the Redis server informed about the new address to use when connecting.
I have a master/slave replication on my MySql DB.
my slave DB was down for a few hours and is back up again (master was up all the time), when issuing show slave status I can see that the slave is X seconds behind the master.
the problem is that the slave dont seem to catch up with the master, the X seconds behind master dont seem to drop...
any ideas on how I can help the slave catch up?
Here is an idea
In order for you to know that MySQL is fully processing the SQL from the relay logs. Try the following:
STOP SLAVE IO_THREAD;
This will stop replication from downloading new entries from the master into its relay logs.
The other thread, known as the SQL thread, will continue processing the SQL statements it downloaded from the master.
When you run SHOW SLAVE STATUS\G, keep your eye on Exec_Master_Log_Pos. Run SHOW SLAVE STATUS\G again. If Exec_Master_Log_Pos does not move after a minute, you can go ahead run START SLAVE IO_THREAD;. This may reduce the number of Seconds_Behind_Master.
Other than that, there is really nothing you can do except to:
Trust Replication
Monitor Seconds_Behind_Master
Monitor Exec_Master_Log_Pos
Run SHOW PROCESSLIST;, take note of the SQL thread to see if it is processing long running queries.
BTW Keep in mind that when you run SHOW PROCESSLIST; with replication running, there should be two DB Connections whose user name is system user. One of those DB Connections will have the current SQL statement being processed by replication. As long as a different SQL statement is visible each time you run SHOW PROCESSLIST;, you can trust mysql is still replicating properly.
What binary log format are you using ? Are you using ROW or STATEMENT ?
SHOW GLOBAL VARIABLES LIKE 'binlog_format';
If you are using ROW as a binlog format make sure that all your tables has Primary or Unique Key:
SELECT t.table_schema,t.table_name,engine
FROM information_schema.tables t
INNER JOIN information_schema .columns c
on t.table_schema=c.table_schema
and t.table_name=c.table_name
and t.table_schema not in ('performance_schema','information_schema','mysql')
GROUP BY t.table_schema,t.table_name
HAVING sum(if(column_key in ('PRI','UNI'), 1,0)) =0;
If you execute e.g. one delete statement on the master to delete 1 million records on a table without a PK or unique key then only one full table scan will take place on the master's side, which is not the case on the slave.
When ROW binlog_format is being used, MySQL writes the rows changes to the binary logs (not as a statement like STATEMENT binlog_format) and that change will be applied on the slave's side row by row, which means a 1 million full table scan will take place on the slave's to reflect only one delete statement on the master and that is causing slave lagging problem.
"seconds behind" isn't a very good tool to find out how much behind the master you really is. What it says is "the query I just executed was executed X seconds ago on the master". That doesn't mean that you will catch up and be right behind the master the next second.
If your slave is normally not lagging behind and the work load on the master is roughly constant you will catch up, but it might take some time, it might even take "forever" if the slave is normally just barely keeping up with the master. Slaves operate on one single thread so it is by design much slower than the master, also if there are some queries that take a while on the master they will block replication while running on the slave.
Just check if you have same time and timezones on both the servers, i.e., Master as well as Slave.
If you are using INNODB tables, check that you have innodb_flush_log_at_trx_commit to a value different that 0 at SLAVE.
http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
We had exactly the same issue after setting up our slave from a recent backup.
We had changed the configuration of our slave to be more crash-safe:
sync_binlog = 1
sync_master_info = 1
relay_log_info_repository = TABLE
relay_log_recovery = 1
I think that especially the sync_binlog = 1 causes the problem, as the specs of this slave is not so fast as in the master. This config option forces the slave to store every transaction in the binary lo before they are executed (instead of the default every 10k transactions).
After disabling these config options again to their default values I see that the slave is catching up again.
Just to add the findings in my similar case.
There were few bulk temporary table insert/update/delete were happening in master which occupied most of the space from relay log in slave. And in Mysql 5.5, since being single threaded, CPU was always in 100% and took lot of time to process these records.
All I did was to add these line in mysql cnf file
replicate-ignore-table=<dbname>.<temptablename1>
replicate-ignore-table=<dbname>.<temptablename2>
and everything became smooth again.
Inorder to figure out which tables are taking more space in relay log, try the following command and then open in a text editor. You may get some hints
cd /var/lib/mysql
mysqlbinlog relay-bin.000010 > /root/RelayQueries.txt
less /root/RelayQueries.txt
If u have multiple schema's consider using multi threaded slave replication.This is relatively new feature.
This can be done dynamically without stopping server.Just stop the slave sql thread.
STOP SLAVE SQL_THREAD;
SET GLOBAL slave_parallel_threads = 4;
START SLAVE SQL_THREAD;
I have an issue similar to this. and both of my MySQL server hosted on AWS EC2 (master and replication). by increasing EBS disk size (which automatically increased IOPS) for MySQL slave server, its turned out the solution for me. R/W Throughput and bandwidth is increased R/W latency were decreased.
now my MySQL database replication is catching up to the master. and Seconds_Behind_Master was decreased (it was got increased from day to day).
so if you have MySQL hosted on EC2. I suggest you tried to increase EBS disk size or its IOPS on the slave.
I know it's been a while since OP asked but it would have helped me to read the following answer.
In /etc/mysql/mysql.cnf :
[mysql]
disable_log_bin
innodb_flush_log_at_trx_commit=2
innodb_doublewrite = 0
sync_binlog=0
disable_log_bin REALLY carried the trick for me.