I recently had a RDS read replica crash, so needed to spin up a new read replica.
Now I have both read replicas there.
I want to delete the one that failed.
Is it just a case of terminating it? Does it have any affect on the master or the other read replica?
Have not ever had to do this before, so want to be sure I am doing this the right way
This should be a perfectly safe operation, but taking a last-minute snapshot of your master (or even the other replica, if it's on-line, replicating, and synched) wouldn't hurt.
In MySQL replication, the connection between master and replica is initiated from the replica ("slave") server side -- if a replica goes offline, the master server just closes the connection and goes about its business; there is essentially no risk to the master from a replica disappearing, even at random.
Specifically, in RDS, replicas seem fairly clearly to be designed to be disposable, and RDS also provides the ability to "promote" a replica -- which is not a promotion to being "the" master (replacing the existing one), but rather a promotion to being "a" master, an independent standalone system of its own, fully disconnected from its former relationship with its former master.
If it made you more comfortable, you could "promote" the replica first, and then delete it once you see that process complete, though simply deleting the replica should be harmless.
http://aws.amazon.com/about-aws/whats-new/2012/10/11/amazon-rds-mysql-rr-promotion/
Related
Weird question I know. I have a master MySQL database which I'm not allowed to touch and need to build a slave for. I would like the slave to be as real time (as possible) of a replica of the master and would like the slave to be HA.
Does MySQL (or MariaDB) replication work when run on a cluster, say, can I make a Galera cluster and make it replicate from a master out of the box or must I use binlog-esque tools?
For the curious; this new slave cluster will be on a different network and will have many large, important queries made against it regularly - the aim of the game is to reduce load on the master and reduce network traffic.
If you are not planning on doing modifications to the downstream slave server, then you can just set up multiple slave servers. This way if one of the slaves goes down you can use another one. This will place a small load on the master for each added slave but whether this added load is even measurable depends on your setup.
Galera could work but I believe you would have to reconfigure one of the nodes to act as the slave if the current one goes down. This would place a minimal load on the master but it would require a manual intervention whenever the current "slave" node goes down.
Parallel replication should also help speed up replication for MariaDB servers.
I have a redis master setup with 2 slaves and a sentinal on each.
If the master goes down for say 2 seconds (+sdown) and comes back up (-sdown), it reads the last snapshot, and the slaves resync with the master.
The problem with this is that there may have been many writes between the last save and the +sdown. It seems like if the master goes from +sdown to -sdown and never +odown (where a failover is initiated), it should be able to sync FROM a slave. My reasoning is that the replication stream is continuous and the slaves most likely have a more accurate reflection of the masters state when +sdown happened.
Is there some config that I can do this? Am I forced to rely on the AOF or snapshots?
(Edit: adding sentinel tag)
You cannot do a partial failover, either you do it or you don't in terms of promoting the slave to master.
From Redis Sentinel:
Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the other additional slaves are reconfigured to use the new master, and the applications using the Redis server informed about the new address to use when connecting.
I setup an EC2 instance with MySQL on EBS volume and setup another instance which acts as Slave for Replication. The replication set up was fine. My question is about taking snapshots of these volumes. I noticed that the tables need to be locked for the snapshot process which may cause inconvenience for the users. So, my idea is to leave the Master instance alone and take a snapshot of instance acting as slave. Is this a good idea? Is there anyone out with a similar setup and could guide me in a right way?
Also, taking snapshot of slave instance would require locking of tables. Would that mean replication will break?
Thanks in advance.
Though it's a good idea to lock the database and freeze the file system when you initiate the snapshot, the actual API call to initiate the snapshot takes a fraction of a second, so your database and file system aren't locked/frozen for long.
That said, there are a couple other considerations you did not mention:
When you attempt to create the lock on the database, it might need to wait for other statements to finish before the lock is granted. During this time, your pending lock might further statements to wait until you get and release the lock. This can cause interruptions in the flow of statements on your production database.
After you initiate the creation of the snapshot, your application/database is free to use the file system on the volume, but if you have a lot of writes, you could experience high iowait, sometimes enough to create a noticeable slowdown of your application. The reason for this is that the background snapshot process needs to copy a block to S3 before it will allow a write to that block on the active volume.
I solve the first issue by requesting a lock and timing out if it is not granted quickly. I then wait a bit and keep retrying until I get the lock. Appropriate timeouts and retry delay may vary for different database loads.
I solve the second problem by performing the frequent, consistent snapshots on the slave instead of the master, just as you proposed. I still recommend performing occasional snapshots against the master simply to improve its intrinsic durability (a deep EBS property) but those snapshots do not need to be performed with locking or freezing as you aren't going to use them for backups.
I also recommend the use of a file system that supports flushing and freezing (XFS). Otherwise, you are snapshotting locked tables in MySQL that might not yet even have all their blocks on the EBS volume yet or other parts of the file system might be modified and inconsistent in the snapshot.
If you're interested, I've published open source software that performs the best practices I've collected related to creating consistent EBS snapshots with MySQL and XFS (both optional).
http://alestic.com/2009/09/ec2-consistent-snapshot
To answer your last question, locking tables in the master will not break replication. In my snapshot software I also flush the tables with read lock to make sure that everything is on the disk being snapshotted and I add the keyword "LOCAL" so that the flush is not replicated to any potential slaves.
You can definitely take a snapshot of the slave.
From your description, it does not seem like the slave is being used operationally.
If this is the case, then the safest method of obtaining a reliable volume snapshot would be to:
Stop mysql server on the slave
start the snapshot (either through the AWS Console, or by command line)
When the snapshot is complete, restart mysqld on the slave server
I read some article about how to implement high-availability solution,the use Heartbeat to check the master mysql is broken,if it broken then switch to the backup server,
one question is when the master server broken,the backup server work on ,how the backup server get the master server 's pre-data
another question is maybe sometimes the master server is not broken,but the mysql service is broken,in this situation is Heartbeat still will switch to the backup server?
What you usually want to do is setup a master-master configuration, but generally only use one master. That was each master also acts as a slave to the other master. In theory, you can modify records in either one and they will keep in sync. In practice, I wouldn't do heavy updates on both servers at once.
My current setup is a master-master configuration, with each master also having a slave off of it. Then you can "failover" to the secondary master/slave and "fail back" when needed. I actually do this if I have database modifications that will take a while.
Basically I want to setup a replication server for mysql datbase. I am completely new to this concept and appreciate any help pointing me in the right direction.
If at all the slave goes down, will it effect the master in anyway?
Thanks.
No, it will not affect the master if the slave goes down. The Slaves connect to the master and request the changes.
If you intend on having 2 servers replicated, then you can use Master-Master replication. This means that either one of the database servers can go down without loss of data or access. This is extremely resilient to failure however you can get duplicate key errors on fast successive inserts, like using MySQL to handle sessions. This can be solved programatically through.
The downside of the Master-Slave set up is that if the master fails you have to manually assign another Master, fix the failed master and then bring back into the group. Otherwise failure of any or all Slaves will not affect the Master.