We are using MySQL version 5.0 and most of the tables are InnoDB. We run replication to a slave server. We are thinking of backing up the MySQL log files on a daily basis.
Questions
Is there any other way of doing an incremental backup without using the log files?
What are the best practices when doing incremental backups?
AFAIK the only way of doing incremental backups is by using the binary-log. You have other options if you want to do full backups (InnoDB hotcopy), but incremental means that you need to log all transactions made.
You need to ask yourself why you're backing up data. Since you have a slave for replication, I assume the backup is primarly for reverting data in case of accidental deletion?
I would probably rotate the logs every 1 hour and take a backup of it. Meaning, restoring would leave the data at most 1 hour old, and you can restore to any point in time since the last full snapshot.
You can dump your schemas regularly with mysqldump, using always the same file name and path for each schema (i.e. replacing the latest one)
Then combine that with any backup tool that supports incremental/delta backup, for example rdiff-backup, duplicity, Duplicati or Areca Backup. An example from duplicity docs:
Because duplicity uses librsync, the incremental archives are space
efficient and only record the parts of files that have changed since
the last backup
That way your first backup would be the compressed copy of the 1st full dump, and the second would contain the compressed differences from the 1st and 2nd dump and so on. You can restore the mysqldump file of any point in time and then restore that file into MySQL.
A lot of time has passed since the last answer, and during this time several solutions and tools have appeared for implementing incremental backups.
Two main ones:
Percona XtraBackup - is an open-source hot backup utility for MySQL -
based servers that doesn’t lock your database during the backup. It
also allows you to create incremental backups. More details here.
It's pretty simple and looks something like this:
xtrabackup --backup --target-dir=/data/backups/inc1 --incremental-basedir=/data/backups/base
mysqlbackup is an utility, that is included in the mysql enterprise
edition. It is a lot like percona xtrabackup. A detailed comparison
can be found here
It has the parameter --incremental which allows you to make incremental backups. More details here
mysqlbackup --defaults-file=/home/dbadmin/my.cnf --incremental --incremental-base=history:last_backup --backup-dir=/home/dbadmin/temp_dir --backup-image=incremental_image1.bi backup-to-image
These two utilities make physical backups (copy database files), but you can still make logical backups of binlog files.
You can either write a script by yourself or use a ready-made script from github:
macournoyer/mysql_s3_backup
Abhishek-S-Patil/mysql-backup-incremental
There are also paid solutions that are, in fact, beautiful wrappers for these tools:
SqlBak
databasethink
What are the best practices when doing incremental backups?
It all depends on your architecture, the amount of data, the maximum allowable downtime interval that is acceptable for you. The maximum allowable data loss interval. Consider these things before setting up backups.
I would like to mention only one good practice, but very important, and which is very often forgotten. Test and run the recovery script regularly on another unrelated server.
Related
I have a mysql cluster database spread on 2 servers.
I want to create a backup system for this database based on the following requirements:
1. Recovery/restore should be very easy and quick. Even better if i can switch connection string at any time i like.
the back up must be like snapshots, so I want to keep copies of different day (and maybe keep the latest 7 days for example)
the copy database does not have to be clustered.
The best way to back up a MySQL Cluster is to use the native backup mechanism that gets initiated with the START BACKUP command in the `ndb_mgm.
Backup is easy (just a single command) and relatively quick. Restore is a bit more tricky, but is at least faster and more reliable than using mysqldump. See also:
http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-backup.html
and
http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-programs-ndb-restore.html
2) The backups are consistent snapshots and are distinguishable by an auto-incrementing backup ID, so having several snapshots is easily possible
3) The backup is clustered by default (every data node stores backup files on its own file system), but you should either have the backup directory pointing to a shared file system mount, or copy files form all nodes to a central place once a backup has finished
Assume a number of conventional LAMP-style applications which use MySQL as a back-end to record the 'current durable state' for the applications.
I am interested in establishing an 'audit' of transitions at the database level - and storing them as a log. The idea is that - assuming the MySQL database has been 'dumped' at the beginning of the day, it would be possible to 'replay' transactions against the back-up to recover any state during the working day.... A bit like time-machine for MySQL - I guess.
I have found some documentation about "Audit plugins" which look relevant but leaves me with more questions than answers.
http://dev.mysql.com/doc/refman/5.6/en/writing-audit-plugins.html
Essentially, I'd like to establish if it would be feasible to write a MySQL plugin to achieve my goal - such that it would work 'seamlessly' with existing MySQL applications?
The principal detail I'm finding it difficult to ascertain is this: When the audit-plugin is notified of an event, what is the mechanism by which the new data can be established in order to log it? How are data types encoded? How hard would it be to write a tool to 'replay' this audit against a 'full-system-backup' using mysqldump, for example?
Are there any existing examples of such plugins?
You just want MySQL's Point-in-Time (Incremental) Recovery Using the Binary Log:
Point-in-time recovery refers to recovery of data changes made since a given point in time. Typically, this type of recovery is performed after restoring a full backup that brings the server to its state as of the time the backup was made. (The full backup can be made in several ways, such as those listed in Section 7.2, “Database Backup Methods”.) Point-in-time recovery then brings the server up to date incrementally from the time of the full backup to a more recent time
Does it make sense to simply copy the mysql\data files vs mysql replication between data centers? I am having the impression mysql replication might be complex when done cross data center. And if I just copy, I could easily switch to the other data center w/o worrying if it's primary or slave. Any thoughts?
MySQL with InnoDB storage engine uses multiversioning of the rows. This means there are changes to the database that may not be yet commited (and possibli will be reverted!). If you simply copy the files, you will end up in inconsistent state.
If you are using MyISAM, copying the files is safe. Replication hovewer will transfer only the changes, while copying will transfer the entire database each time. Which is not wise with large databases.
Replication synchronizes database between data centers "live".
While coping whole database takes a lot of time and databases will desynchronize after first change is made.
Without not reason, I lose all my data in my database. Fortunately this was just test data, but this made me to think what will happen if this was done with a production db.
Ultimately, every developer got a db problem and want to rollback the db. We don't do things to protect the db, as we think its a DBA work, but then we got into trouble...
What are your backup best practices?
Since all the developers are also the DBAs where I work, we're collectively responsible for our backup strategy as well - if you care about the data, make sure you're at least informed about how the backups work, even if you're not part of the actual decisions.
The VERY first thing I do (before I even have any databases set up) is set up nightly maintenance plans that include a full backup, and direct those backups to a central network share on a different computer (our NAS). At the very least, for the love of your job, don't put the backups on the same physical storage that your database files sit on. What good are backups if you lose them at the same time you lose the disk?
We don't do point-in-time restores, so we don't do log backups (all our databases are set to Simple recovery mode), but if you want logs backed up, make sure you include those as well, as an acceptable interval as well.
On a side note, SQL 2008 supports compressed backups, which speeds up backup time considerably and makes the files much, much smaller - I can't think of an instance where you wouldn't want to use this option. I'd love to hear one, though, and I'm willing to reconsider!
Here are some points from my own experience:
Store your backups and SQL Server database files on the different physical storages. Otherwise, if your physical storage failed you will lose both backups and database files.
Create your own SQL Server database backup schedule.
Test your SQL Server backup. If you never tested your backups I have a doubt that you will be able to restore your database if the failure occurs. Time to time you need to have the practice to restore your backup on a test server.
Test your recovery strategy - here is another tip. If the failure occurs how much time do you need to restore your database to the working state?
Backup SQL Server's system databases.
And this isn't a whole list, you can find more tips in my article https://sqlbak.com/blog/backup-and-recovery-best-practices/
Choosing the right backup strategy is one of the most important factors a DBA should consider at the point of developing the DB.
However the backup strategy you choose depends on a number of factors:
How frequent is transaction carried out on the DB: are there thousands of transactions going on every minute or maybe few transactions per day? For very busy database, i would say, take full nightly backups and transaction log backups every 10 mins or even less.
How critical the data content is: may it be employee payroll data? Then you'll have no acceptable excuse or you may have a few angry faces around your car when you want to drive home! For very critical database, take nightly backups possibly in two locations and transaction log backups every 5 mins. (Also think of implementing mirroring).
Location of your backup location: if your backup location is close to the DB location, then you can afford to take more frequent backup than when it is located several hops away with not excellent bandwidth between.
But in all, i would say, schedule a nighly backup every night, and then transaction log backups at intervals.
The application that I am working on generates files dynamically with use. This makes backup and syncronization between staging,development and production a real big challenge. One way that we might get smooth solution (if feasable) is to have a script that at the moment of backing up the database can backup the dynamically generated files inside the database and in restore time can bring those file out of the database and in the filesystem again.
I am wondering if there are any available (pay or free) application that could be use as scripts to make this happen.
Basically if I have
/usr/share/appname/server/dynamicdir
/usr/share/appname/server/otherdir/etc/resource.file
Then taking the examples above and with the script put them on the mysql database.
Please let me know if you need more information.
Do you mean that the application is storing a files as blobs in the MySQL database, and/or creating lots of temporary tables? Or that you just want temporary files - themselves unrelated to a database - to be stored in MySQL as a backup?
I'm not sure that trying to use MySQL as an net-new intermediary for backups of files is a good idea. If the app already uses it, thats one thing, if not, MySQL isn't the right tool here.
Anyway. If you are interested in capturing a filesystem at point-in-time, the answer is to utilize LVM snapshots. You would likely have to rebuild your server to get your filesystems onto LVM, and have enough free storage there for as many snapshots as you think you'd need.
I would recommend having a new mount point just for this apps temporary files. If your MySQL tables are using InnoDB, a simple script to run mysqldump --single-transaction in the background, and then the lvm snapshot process, you could get these synced up to less then a second.
the should be trivial to accomplish using PHP, perl, python, etc. are you looking for someone to write this for you?