I have a mysql cluster database spread on 2 servers.
I want to create a backup system for this database based on the following requirements:
1. Recovery/restore should be very easy and quick. Even better if i can switch connection string at any time i like.
the back up must be like snapshots, so I want to keep copies of different day (and maybe keep the latest 7 days for example)
the copy database does not have to be clustered.
The best way to back up a MySQL Cluster is to use the native backup mechanism that gets initiated with the START BACKUP command in the `ndb_mgm.
Backup is easy (just a single command) and relatively quick. Restore is a bit more tricky, but is at least faster and more reliable than using mysqldump. See also:
http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-backup.html
and
http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-programs-ndb-restore.html
2) The backups are consistent snapshots and are distinguishable by an auto-incrementing backup ID, so having several snapshots is easily possible
3) The backup is clustered by default (every data node stores backup files on its own file system), but you should either have the backup directory pointing to a shared file system mount, or copy files form all nodes to a central place once a backup has finished
Related
I want to create a Duplicate (clone) of an Aurora DB cluster.
Both the source and the copy are in the same region and are both for dev purposes .
Both are MySql.
I want to access each cluster via a different url .
Reading about Copy-on-write protocol for Aurora cloning.
and SQL snapshot
The aws docs state that :"Creating a clone is faster and more space-efficient than physically copying the data using a different technique such as restoring a snapshot." (source)
Yet , I don't quite understand why using a snapshot is an inferior solution ?
Snapshot is slower, because first snapshot copies entire db storage:
The amount of time it takes to create a DB cluster snapshot varies with the size your databases. Since the snapshot includes the entire storage volume, the size of files, such as temporary files, also affects the amount of time it takes to create the snapshot.
So if you database has, lets say 100GB, the first snapshot of it will require copying 100 GBs. This operation can take time.
In contrast, when you clone, there is no copy done at first. Both original and the new database use same storage. Only when a write operation is performed, they start to diverge.
I am looking into migrating my MySQL DB to Azure Database for MySQL https://azure.microsoft.com/en-us/services/mysql/. It currently resides on a server hosted by another company. The DB is about 100 GB. (It worries me that Azure uses the term "relatively large" for 1GB.)
Is there a way to migrate the DB without any or little (a few hours, max) downtime? I obviously can't do a dump and load as the downtime could be days. Their documentation seems to be for syncing with a MySQL server that is already on a MS server.
Is there a way to export the data out of MS Azure if I later want to use something else, again without significant downtime?
Another approach: Use Azure Data Factory to copy the data from your MySQL source to your Azure DB. Set up a sync procedure that updates your Azure Database with new rows. Sync, take MYSQL db offline, sync once more and switch to the Azure DB.
See Microsoft online help
Don't underestimate the complexity of this migration.
With 100GB, it's a good guess that most rows in your tables don't get UPDATEd or DELETEd.
For my suggestion here to work, you will need a way to
SELECT * FROM table WHERE (the rows are new or updated since a certain date)
Some INSERT-only tables will have autoincrementing ID values. In this case you can figure out the ID cutoff value between old and new. Other tables may be UPDATEd. Unless those table have timestamps saying when they were updated, you'll have a challenge figuring it out. You need to understand your data to do that. It's OK if your WHERE (new or updated) operation takes some extra rows that are older. It's NOT OK if it misses INSERTed or UPDATEd rows.
Once you know how to do this for each large table, you can start migrating.
Mass Migration Keeping your old system online and active, you can use mysqldump to migrate your data to the new server. You can take as long as you require to do it. Read this for some suggestions. getting Lost connection to mysql when using mysqldump even with max_allowed_packet parameter
Then, you'll have a stale copy of the data on the new server. Make sure the indexes are correctly built. You may want to use OPTIMIZE TABLE on the newly loaded tables.
Update Migration You can then use your WHERE (the rows are new or updated) queries to migrate the rows that have changed since you migrated the whole table. Again, you can take as long as you want to do this, keeping your old system online. It should take much less time than your first migration, because it will handle far fewer rows.
Final Migration, offline Finally, you can take your system offline and migrate the remaining rows, the ones that changed since your last migration. And migrate your small tables in their entirety, again. Then start your new system.
Yeah but, you say, how will I know I did it right?
For best results, you should script your migration steps, and use the scripts. That way your final migration step will go quickly.
You could rehearse this process on a local server on your premises. While 100GiB is big for a database, it's not an outrageous amount of disk space on a desktop or server-room machine.
Save the very large extracted files from your mass migration step so you can re-use them when you flub your first attempts to load them. That way you'll save the repeated extraction load on your old system.
You should stand up a staging copy of your migrated database (at your new cloud provider) and test it with a staging copy of your application. You may be able to do this with a small subset of your rows. But do test your final migration step with this copy to make sure it works.
Be prepared for a fast rollback to the old system if the new one goes wrong .
AND, maybe this is an opportunity to purge out some old data before you migrate. This kind of migration is difficult enough that you could make a business case for extracting and then deleting old rows from your old server, before you start migrating.
Does it make sense to simply copy the mysql\data files vs mysql replication between data centers? I am having the impression mysql replication might be complex when done cross data center. And if I just copy, I could easily switch to the other data center w/o worrying if it's primary or slave. Any thoughts?
MySQL with InnoDB storage engine uses multiversioning of the rows. This means there are changes to the database that may not be yet commited (and possibli will be reverted!). If you simply copy the files, you will end up in inconsistent state.
If you are using MyISAM, copying the files is safe. Replication hovewer will transfer only the changes, while copying will transfer the entire database each time. Which is not wise with large databases.
Replication synchronizes database between data centers "live".
While coping whole database takes a lot of time and databases will desynchronize after first change is made.
i has two ubuntu server,and installed Mysql 4.0, ServerA and ServerB
i want to synchronize ServerA's mysql table to ServerB (copy ServerATable to ServerBTable, keep ServerATable and ServerBTable equals)
how to do it?
(shell script,or java, or linux mysql tools)
thanks :)
There's a few options. One is to set up replication within MySQL, which will automatically copy data back and forth between the servers and keep them synchronized automatically, with a granularity of a few seconds as the data flies around. Downside is you have to expose at least the master server to the net to allow TCP connections.
Externally you can do regular mysqldumps on server A, copy to server B, load into mysql, and off you go. This will have a granularity of whatever time interval you run the dump/copy/load sequence in. Downside is that mysqldump locks tables as it's working. If you have a large database, you'll be locked out of serverA's tables while the dump progresses, and the locked out of serverB as the data's loaded. And loading a dump is much slower than doing the dump in the first place, as mysql does all kinds of internal housekeeping (key updates, table metadata updates, etc...) during the load.
For the third option, you could write a synchronizer that compares rows between the two databases and does updates as necessary. However, then you're slurping the contents of one server over to the other and doing row-by-row operations. Plus having to handle any dependent foreign key/child relation updates as well, and this will get complicated in a hurry. At that point you might be better off using the mysqldump option instead.
Of course, you should upgrade your MySQL to something more modern. v4.x is very outdated. You should be running at least a 5.0, preferably a 5.1 version instead.
We are using MySQL version 5.0 and most of the tables are InnoDB. We run replication to a slave server. We are thinking of backing up the MySQL log files on a daily basis.
Questions
Is there any other way of doing an incremental backup without using the log files?
What are the best practices when doing incremental backups?
AFAIK the only way of doing incremental backups is by using the binary-log. You have other options if you want to do full backups (InnoDB hotcopy), but incremental means that you need to log all transactions made.
You need to ask yourself why you're backing up data. Since you have a slave for replication, I assume the backup is primarly for reverting data in case of accidental deletion?
I would probably rotate the logs every 1 hour and take a backup of it. Meaning, restoring would leave the data at most 1 hour old, and you can restore to any point in time since the last full snapshot.
You can dump your schemas regularly with mysqldump, using always the same file name and path for each schema (i.e. replacing the latest one)
Then combine that with any backup tool that supports incremental/delta backup, for example rdiff-backup, duplicity, Duplicati or Areca Backup. An example from duplicity docs:
Because duplicity uses librsync, the incremental archives are space
efficient and only record the parts of files that have changed since
the last backup
That way your first backup would be the compressed copy of the 1st full dump, and the second would contain the compressed differences from the 1st and 2nd dump and so on. You can restore the mysqldump file of any point in time and then restore that file into MySQL.
A lot of time has passed since the last answer, and during this time several solutions and tools have appeared for implementing incremental backups.
Two main ones:
Percona XtraBackup - is an open-source hot backup utility for MySQL -
based servers that doesn’t lock your database during the backup. It
also allows you to create incremental backups. More details here.
It's pretty simple and looks something like this:
xtrabackup --backup --target-dir=/data/backups/inc1 --incremental-basedir=/data/backups/base
mysqlbackup is an utility, that is included in the mysql enterprise
edition. It is a lot like percona xtrabackup. A detailed comparison
can be found here
It has the parameter --incremental which allows you to make incremental backups. More details here
mysqlbackup --defaults-file=/home/dbadmin/my.cnf --incremental --incremental-base=history:last_backup --backup-dir=/home/dbadmin/temp_dir --backup-image=incremental_image1.bi backup-to-image
These two utilities make physical backups (copy database files), but you can still make logical backups of binlog files.
You can either write a script by yourself or use a ready-made script from github:
macournoyer/mysql_s3_backup
Abhishek-S-Patil/mysql-backup-incremental
There are also paid solutions that are, in fact, beautiful wrappers for these tools:
SqlBak
databasethink
What are the best practices when doing incremental backups?
It all depends on your architecture, the amount of data, the maximum allowable downtime interval that is acceptable for you. The maximum allowable data loss interval. Consider these things before setting up backups.
I would like to mention only one good practice, but very important, and which is very often forgotten. Test and run the recovery script regularly on another unrelated server.