I have a couple of live databases that need migrating over to a new server. These are large databases constantly in use.
I want to setup replication with a slave on the new server and begin porting data across. However, I'd like to try and avoid doing a mysqldump on the current master data to get the initial binlog position as I don't want to lock down the database for an extended period of time.
Is there a way that I can find out the earliest master_log_pos so I can start replication from the very start? If not, are there any other solutions that avoid halting transactions (as much as possible)?
If I understand you correctly, I think Percona's Xtrabackup is helpful for you.
Xtrabackup is hot backup tool which makes you can backup MySQL while it is running.
With Xtrabackup you can make data backup quickly even if data is large and move backups to slaves.
How to setup a slave for replication in 6 simple steps with Percona XtraBackup is good start point.
http://www.percona.com/doc/percona-xtrabackup/2.1/ is Xtrabackup documentation
Setting up Master-Slave replication using xtrabackup also describes how to use it for making slave.
Related
I've set up a slave replication of MySQL database. And for development requirements, I want to write sth into the slave database, but it would cause the broken of replication.
Since the database is huge, I don't want to restore the slave database from MySQL dump file every time after I finished some development work.
My requirement:
All the changes in the slave database can be reverted by a simple command.
The replication keeps working.
One method is to use LVM filesystem snapshots. Before you begin testing:
Stop replication.
Take an LVM snapshot.
Do your tests. Replication is still off, but data is up to date
After you finish testing:
Stop mysqld.
Restore the snapshot. This reverts all files to the state they were at the moment you created the LVM snapshot above.
Start mysqld and start replication. It will need to catch up and apply all changes since you stopped replication before your testing. This will take a little while, depending on how many changes happened on your master database.
See https://www.tecmint.com/take-snapshot-of-logical-volume-and-restore-in-lvm/ for a nice tutorial on using LVM snapshots.
This method only works if your development database instance is on Linux.
Insert the new records using a primary key that is not expected to be used by the master database (e.g. add a sufficiently large offset like 2^10 or negative numbers if allowed...).
In this way, the insertions coming from the master won't clash.
There are multiple ways to take a MySQL replication, or rebuilding it. How can we build master-slave replications via LVM snapshot?
And also which is the most effective way among below 4 choices & why ??
Setting up MySQL Replication using mysqldump.
Setting up MySQL Replication using Percona XtraBackup.
Setting up MySQL Replication using Rsync.
setting up MySQL Master-Slave Replication with LVM snapshots.
Since setting up LVM requires major surgery at the filesystem level, that approach could be a challenge. However, once it is done, the rest is relatively painless.
XtraBackup is an improvement on mysqldump.
rsync is good for copying files that are not being modified. When MySQL is running, it is modifying files, and caching unwritten stuff in ram. Sounds sketchy.
I am handling few databases that is growing very fast. Now it is around 12GB and in next few months it will cross 15GB. At this situation I only have the traditional backup process mysqldump running in cronjob. I found significant delay in backup and restore time (hours, even days) for the whole database. Knowing that physical backups are much faster than logical, I found these two, but cannot implement those because of some limitations and company policy.
MySQL Enterprise Backup : which is commercial
Percona XtraBackup : Percona came with this tool specially for Linux environments. And not all companies will agree to use 3rd party tools even though it is open source.
Please suggest any other better backup and restore mechanism for big database specially when the databases are mix of Innodb and MyISAM. Looking for some good suggestions on snapshots
I have personally used Percona XtraBackup toolset. It does some simple procedures to get all the committed data in innodb binlog when it finishes the normal storage. Reading about MySQL Enterprise Backup also can do this.
Incremental backups/snapshots can be made with both Percona XtraBackup and MySQL Enterprise, if that's what you're looking for.
Using a read-only slave, you can maintain separation of the read locks. This will keep your locks from being held on the master read if you backup using mysql free tools.
Another option if you have a slave is you can shut off the slave and rsync the data somewhere for backup.
First I would try using "--single-transaction" with mysqldump as long as all your tables are innodb and you aren't modifying table structure during dump. This will attempt to dump from a single point in time but only when it can be performed without locking your tables.
I haven't tried xtrabackup but it looks like that will do what you want as well.
You could also try setting up mysql replication and dumping from the slave.
Lastly LVM snapshots but that takes a bit more work.
Surprisingly I can't find anything on the internet which speaks to this. We currently have a master slave slave mysql set up. We are considering making a move to mysql cluster, for scalability reasons. It would be really nice if we could make the cluster a slave of the current master and run a dev environment off the cluster for now. Even better would be if we could do a master master relationship between the current master and the cluster.
Our concern is that our data has to be kept very current, so not being able to do this replication would mean downtime. If this replication is not possible, what is the smoothest possible way to make the transition. We're currently running mariaDB.
I have a scenario where there are multiple mysql 4.x servers. These databases were supposed to be replicating to another server. After checking things out on a slave it appears that this slave has not replicated any databases in some time.
Some of these databases are > 4G in size and one is 43G(which resides on another server). Has anyone out there replicated databases without creating a snapshot to copy over to a slave? I cannot shutdown the master server because of the downtime. It will probably take over an hour and 40 minutes to create a snapshot. So this is out of the question.
I was going to perform a load data from master on the slave to pull everything from scratch. Any idea how long this will take on databases ranging from 1-4G and the 43G database will be for another day. All of the tables on the master are myIsam so I don't think I will have a problem with the load from master method.
What are the best methods on the slave to clean things up or reset things so I can just start from a clean slate?
Any suggestions?
Thanks in advance
You need a snapshot to start replication. Snapshots require either the database to be locked (at least) read-only. So you can have a consistent place to start from.
Downtime is a necessary thing, customers usually understand it as long as it doesn't happen too often.