I'm currently using mysqldump to back up databases that are growing rapidly in size. Though I run it late at night, there have been occasional problems when it happens to run during a moment of high traffic (which happens at night sometimes). For example, last night one of my sites locked up just after the time of the database backup with a completely full (and non-clearing) processlist.
Does anyone have a suggestion for a better way to approach this? Putting the site in a temporary maintenance state during backup is not an option as the goal is to maximize availability (some sql dumps take awhile). One idea that comes to mind is to run both master and slave copies and shut down + back up the slave copy, leaving the master copy alone during the process. Hopefully there is a simpler solution though - I'd rather not run a slave copy for backup purposes only unless absolutely necessary. Any suggestions?
Thanks.
Two thoughts:
run the slave. If nothing else, it gives you a warm spare for your production traffic in case of failure. You can also run reports and tools from it, freeing up cycles from your production server.
get to innodb and use mysqldump --single-transaction (see man page)
Good luck!
I use Percona Xtrabackup, which is similar to InnoDB Hot Backup with more functionality and is distributed for free. Xtrabackup takes snapshots without locking innodb tables and will record the current master logfile info and, if requested, the slave info if you are taking a backup from a slave.
I would recommend running a slave and doing a backup like this or with mysqldump. The slave gives you a hot backup that you can quickly switch over to and be up and running within minutes if your master blows up due to a hardware issue or various software or user error issues that take out the server. The backup with xtrabackup or mysqldump gives you a backup that you can use to restore data in case you accidentally drop a table or delete some rows you shouldn't have, since the replicated server wouldn't save you there.
Related
We have one php application and mysql server running on one of our production server.
Mysql server is currently 4GB big with intention to grow up to tens or even up to hundreds of GB.
What am curious to find out is what are the best practices for backup of mysql database in condition that application must be live under any circumstance? What is better, to have mysql replication server on which we will run backup scripts or to run on live server? What is more likely to slow down We have possibility to add additional server(s) if needed. Where do I need to store mysql dumps? Is it suggested to ftp copy mysql backup files to remote server.
What is the best practice to organize web application backup if don't have problem with number of server instances?
MySQL backup methods are documented on MySQL documentation.
The ideal backup solution will be to use MySQL Enterprise Backup. This is a licensed product sold on Oracle store. It is very fast compared to mysqldump.
MySQL Enterprise Backup: A licensed product that performs hot backups
of MySQL databases. It offers the most efficiency and flexibility when
backing up InnoDB tables, but can also back up MyISAM and other kinds
of tables.
If you are looking for a free solution with MySQL community edition, then you can install another replication server and either run mysqldump to take backup or make a raw data backup. During backup on your replication server, your main master database will be running. Since your data is big or will get bigger, it is recommended to backup raw data files. It is basically a process of copying data and log files from disk. Details are explained on MySQL documentation.
For larger databases, where mysqldump would be impractical or
inefficient, you can back up the raw data files instead. Using the raw
data files option also means that you can back up the binary and relay
logs that will enable you to recreate the slave in the event of a
slave failure.
Finally, you should copy backup files to another physical disk on the same to recover from disk failures or to another physical server to easily recover from complete server failures.
Replication is something that protects against hardware errors, for example, a hard disk crashed.
Backup - protects against software errors, for example, due to the human factor, data has been deleted from a table.
It is definitely good practice to combine both of these technologies by running a utility to create a backup on a replica. This not only reduces the load on the product database, but also covers more recovery scenarios.
In case of a hardware error, you can restore the most up-to-date data from the replica, and in cases of data corruption, you can already consider about from the what date to use the backup for recovery. Well, if your both the main server and the replica fail, then the backup will also save you.
What is the best way to make backups?
mysqldump is a good solution for small databases. This is a utility for creating logical backups nad it is included to MySQL Server. At the output, the utility creates a .sql file to recreate the database.
For large databases, it is better to use a physical backup. There are two ways on how to do it.
mysqlbackup is a utility included with MySQL Enterprise Solution. As a result, you get a binary file. Such a backup is created much faster than using mysqldump and is less load on the server.
xtrabackup, from Percona, is a lot like the MySQL Enterprise backup utility, but it's free. A more detailed comparison can be found here.
How often the backups should be made?
The more often you make backups, the better, but you can't make many such backups - since you will run out of space in the backup storage. There are two ways:
Find a compromise between the frequency of backups and the duration of storage.
Use incremental backups. The above utilities support incremental backups, but the management of such backups is more complicated (read more here)
Where the backups should be stored?
Anywhere you prefer, but not in the same place as the MySQL Server. Overall, I think using cloud storage is a good choice. Almost everyone today has a command line interface.
How to automate a backup?
The process of creating regular backups should be automated, and a person should intervene in it only in case of failure. A good backup process should include the following steps:
Creating a backup copy
Compression\Encryption
Uploading to storage
Sending success\fail notification
Removing old backups from the storage (so that it does not overflow)
The simplest script that implements this can be found, for example, here.
Something else?
Yes, the most important thing is not to create a backup and then restore it. Therefore, it is best practice to regularly test the recovery scenarios.
Happy backups!
What is better, to have mysql replication server on which we will run backup scripts or to run on live server
It depends on your db size (and time needed to dump it using mysqldump) and your reliability requirements.
If your db is relatively small and mysqldump dumps it in seconds or in a few minutes then its ok to just run scheduled backups. For most cases it is sufficient to have a daily backup which runs at a time when your app is mostly idle (at night when you clients are sleeping). You can use a nice tool automysqlbackup for that: it cares about the scheduling and backup rotation, all you need to do is to add it as your cron task and set up its config once.
Setting up a replica is only needed if:
Your backup takes long time (dozens of minutes or hours) to complete so you can not just stop your service for that long.
You can not afford loosing any history in case of main db crash. E.g. if you process financial transactions you may want to ensure that nothing will be lost if master db server dies.
In this cases you may want a replica with backups. Though you must understand that adding replication adds a new layer of problems: replicas may go out of sync, silently crash (and you will not notice that as the master and your app is running fine) etc.
We've got a database (mysql) driven application which contains business critical information, were looking at building a system that will allow us to backup the db frequently (say every 15 mins) essentially so that we minigate the danger of any data loss. Where torn between two setups :
Adding a backup jobs too a queue every 15 mins on a cron and storing these backups on another server. (To save space we would then delete most of these backups after 3 days, but keep the 06:00, 12:00, 18;00 hour versions.)
or
Is there a RAID like setup were all our data will be automatically copied to another hard drive or in this case server, in which case what would happen if we lost data, would the loss be carried to the other server (we would also run standard daily backups for our archives in edition to this) ?
or
Is there another established method for creating frequent backups ?
In my opinion, the optimal backup scheme would be following.
Delayed slave. It allows you quickly restore your database in case of master failure. It may help in case of DROP DATABASE or other wrong SQL. So, you need something additionally.
Incremental backups every day with Xtrabackup from the delayed slave. Optionally you could also check TwinDB for incremental backups.
As long as you need 15 minutes granularity you may pull binary logs from the master with mysqlbinlog from MySQL 5.6 (even if the master is 5.5 or 5.1). So, mysqlbinlog runs on a remote host and pulls logs from the master.
If you need to restore the database you have two ways.
If you can restore from the delayed slave you use that slave as a new master.
If on some reason you can't use the delayed slave (you missed the DROP command) then you restore last night copy from the incremental backup and apply binary logs since the last backup up to the moment of accident (again, if the accident is wrong DROP table you replay logs up to the last event before the DROP).
This schema will be optimal from performance standpoint (no impact on an application) and allows no data loss at all.
If you're doing backups more often than one hour, what you need is replication. Setting up a secondary database server that can serve as a hot-standby is a lot better than abusing your database with repeated reads.
If you're backing up your database frequently, look at innobackupex to snapshot your tables, or possibly LVM snapshots.
Currently my database is almost 20 GB big and still growing.
I'm taking a daily backup with mysqldump and it's getting really slow.
So slow that in the meanwhile new connections stack up and eventually cause this error:
SQLSTATE[HY000] [1040] Too many connections
(I could improve the amount of connections that's accepted but that won't do anything because the connections are still just frozen, waiting for the backup to complete, which will lead to timeout)
I've been reading up on some options to improve the speed and this is what I've found:
option --quick (Will probably help)
option --single-transaction (Will prevent tables from being locked, but may cause database to become incorrect)
Master-Slave replication (Probably the best thing I could do, one problem, I have only one server available)
The master-slave replication really sounds like it's the best option since I can stop the slave from updating, take the backup, and let it resume syncing. The problem is I only have one fysical machine to work with.
I know that I can set up multiple mysql instances on this one server. The question is: Is it wise to do so?
The slave is really only used to generate that backup file (which will be copied to a different disk on the network) so that the master can stay live.
if you use just innodb - try xtrabackup.
if you use both myisam and innodb - flush + lvm snapshot + file-level copy might work for you.
indeed replication slave for backups is good idea as well. just remember to periodically check data consistency between the master and the slave.
I maintain big MySQL database. I need to backup it every night, but the DB is active all the time. There are queries from users.
Now I just disable the website and then do a backup, but this is very bad as the service is disabled and users don't like this.
What is a good way to backup the data if data is changed during the backup?
What is best practice for this?
I've implemented this scheme using a read-only replication slave of my database server.
MySQL Database Replication is pretty easy to set up and monitor. You can set it up to get all changes made to your production database, then take it off-line nightly to make a backup.
The Replication Slave server can be brought up as read-only to ensure that no changes can be made to it directly.
There are other ways of doing this that don't require the replication slave, but in my experience that was a pretty solid way of solving this problem.
Here's a link to the docs on MySQL Replication.
If you have a really large (50G+ like me) MySQL MyISAM only databases, you can use locks and rsync. According to MySQL documentation you can safely copy raw files while read lock is active and you cannot do it with InnoDB.
So if the goal is zero downtime and you have extra HD space, create a script:
rsync -aP --delete /var/lib/mysql/* /tmp/mysql/sync
Then do the following:
Do flush tables
Run script
Do flush tables with read lock;
Run script again
Do unlock tables;
On first run rsync will copy a lot without stopping MySQL. The second run will be very short, it will only delay write queries, so it is a real zero downtime solution.
Do another rsync from /tmp/mysql/sync to a remote server, compress, keep incremental versions, anything you like.
This partly depends upon whether you use innodb or myiasm. For innodb; mySQL have their own (which costs money) solution for this (innodb hot copy) but there is an open source version from Percona you may want to look at:
http://www.percona.com/doc/percona-xtrabackup/
What you want to do is called "online backup". Here's a pointer to a matrix of possible options with more information:
http://www.zmanda.com/blogs/?p=19
It essentially boils down to the storage backend that you are using and how much hardware you have available.
I have a scenario where there are multiple mysql 4.x servers. These databases were supposed to be replicating to another server. After checking things out on a slave it appears that this slave has not replicated any databases in some time.
Some of these databases are > 4G in size and one is 43G(which resides on another server). Has anyone out there replicated databases without creating a snapshot to copy over to a slave? I cannot shutdown the master server because of the downtime. It will probably take over an hour and 40 minutes to create a snapshot. So this is out of the question.
I was going to perform a load data from master on the slave to pull everything from scratch. Any idea how long this will take on databases ranging from 1-4G and the 43G database will be for another day. All of the tables on the master are myIsam so I don't think I will have a problem with the load from master method.
What are the best methods on the slave to clean things up or reset things so I can just start from a clean slate?
Any suggestions?
Thanks in advance
You need a snapshot to start replication. Snapshots require either the database to be locked (at least) read-only. So you can have a consistent place to start from.
Downtime is a necessary thing, customers usually understand it as long as it doesn't happen too often.