I am looking for a hot backup system for mysql databases? I like to ask you for experiences and expertise.
So the place you want to look for good solutions to MySQL issues is Mysql Performance Blog/Percona. Specifically, what you want to do is use their XtraBackup, which fixes a number of issues that occur with InnoDBs hot backup. You'd be well served to use their patched server with their InnoDB replacement XtraDB.
They integrate a number of third party patches for MySQL, including their own and Google's, and have enhanced the InnoDB storage engine with fixes, enhancements and better logging. Additionally, Percona sponsors a fantastic toolbox for MySQL called Maatkit, which gives some components to help you with backups and integrity checks.
I know InnoDB hot backup but works just for InnoDB
http://www.innodb.com/products/hot-backup/
I use the MySQL GUI Administrator Tool. It has a backup tool, which you can use to schedule daily backups through windows. You can create different backup projects, schedule the backups, and control how the backup runs and the type of output file it creates.
I've always found it reliable, and the other tools bundled with it are also useful for basic database administration.
Related
I am designing a data replication solution across timezones and have run into the issue where I can only run and old version of MySQL (5.6) in one location, whereas the other two have MariaDB 10.2.
Now, I have read the information about Replication Compatibility over at MariaDB. Clearly I can't use MariaDB as a master and MySQL as a slave.
Intermediate solution based on Bash scripts
Yet, I want to use my EU server as a master and that is running MariaDB. So I'm now contemplating a way around the limitation. So far I have come up with an intermediate data storage solution in the overseas server, where data is shuffled periodically using my own Bash data migration scripts over a low bandwidth link.
MariaDB is required in the primary location
I have to use MariaDB in my primary location because I'm using the ColumnStore database there. That is unconditional as part of the application design.
Does this situation ring a bell?
Do you have experience with similar situations and would you mind sharing some inspiration as for how you did it?
My best solution so far is with Bash scripts that are cronned, where MySQL data is dumped (mysqldump) and transferred over a low bandwidth link, then merged with the master (ColumnStore storage engine). I'm looking at a T+1 data lag between my primary location and the secondary location that is running MySQL.
Any high level design thoughts or shared experience is highly appreciated.
Best regards
I am planning to migrate a large MySQL database from a Windows Machine to a Linux machine. The databases uses the InnoDB storage engine on the tables. Because mysqldump is so slow with large databases, i want to use raw data (files) to perform the task. I understand that when doing it this way, the hardware on both machines and the MySQL versions have to be identical, otherwise its complex to accomplish. Now my question is - Is there perhaps a tool that i can use to help accomplish this...maybe a tool that requires to pay or even free it does not matter? Please advise.
We have one php application and mysql server running on one of our production server.
Mysql server is currently 4GB big with intention to grow up to tens or even up to hundreds of GB.
What am curious to find out is what are the best practices for backup of mysql database in condition that application must be live under any circumstance? What is better, to have mysql replication server on which we will run backup scripts or to run on live server? What is more likely to slow down We have possibility to add additional server(s) if needed. Where do I need to store mysql dumps? Is it suggested to ftp copy mysql backup files to remote server.
What is the best practice to organize web application backup if don't have problem with number of server instances?
MySQL backup methods are documented on MySQL documentation.
The ideal backup solution will be to use MySQL Enterprise Backup. This is a licensed product sold on Oracle store. It is very fast compared to mysqldump.
MySQL Enterprise Backup: A licensed product that performs hot backups
of MySQL databases. It offers the most efficiency and flexibility when
backing up InnoDB tables, but can also back up MyISAM and other kinds
of tables.
If you are looking for a free solution with MySQL community edition, then you can install another replication server and either run mysqldump to take backup or make a raw data backup. During backup on your replication server, your main master database will be running. Since your data is big or will get bigger, it is recommended to backup raw data files. It is basically a process of copying data and log files from disk. Details are explained on MySQL documentation.
For larger databases, where mysqldump would be impractical or
inefficient, you can back up the raw data files instead. Using the raw
data files option also means that you can back up the binary and relay
logs that will enable you to recreate the slave in the event of a
slave failure.
Finally, you should copy backup files to another physical disk on the same to recover from disk failures or to another physical server to easily recover from complete server failures.
Replication is something that protects against hardware errors, for example, a hard disk crashed.
Backup - protects against software errors, for example, due to the human factor, data has been deleted from a table.
It is definitely good practice to combine both of these technologies by running a utility to create a backup on a replica. This not only reduces the load on the product database, but also covers more recovery scenarios.
In case of a hardware error, you can restore the most up-to-date data from the replica, and in cases of data corruption, you can already consider about from the what date to use the backup for recovery. Well, if your both the main server and the replica fail, then the backup will also save you.
What is the best way to make backups?
mysqldump is a good solution for small databases. This is a utility for creating logical backups nad it is included to MySQL Server. At the output, the utility creates a .sql file to recreate the database.
For large databases, it is better to use a physical backup. There are two ways on how to do it.
mysqlbackup is a utility included with MySQL Enterprise Solution. As a result, you get a binary file. Such a backup is created much faster than using mysqldump and is less load on the server.
xtrabackup, from Percona, is a lot like the MySQL Enterprise backup utility, but it's free. A more detailed comparison can be found here.
How often the backups should be made?
The more often you make backups, the better, but you can't make many such backups - since you will run out of space in the backup storage. There are two ways:
Find a compromise between the frequency of backups and the duration of storage.
Use incremental backups. The above utilities support incremental backups, but the management of such backups is more complicated (read more here)
Where the backups should be stored?
Anywhere you prefer, but not in the same place as the MySQL Server. Overall, I think using cloud storage is a good choice. Almost everyone today has a command line interface.
How to automate a backup?
The process of creating regular backups should be automated, and a person should intervene in it only in case of failure. A good backup process should include the following steps:
Creating a backup copy
Compression\Encryption
Uploading to storage
Sending success\fail notification
Removing old backups from the storage (so that it does not overflow)
The simplest script that implements this can be found, for example, here.
Something else?
Yes, the most important thing is not to create a backup and then restore it. Therefore, it is best practice to regularly test the recovery scenarios.
Happy backups!
What is better, to have mysql replication server on which we will run backup scripts or to run on live server
It depends on your db size (and time needed to dump it using mysqldump) and your reliability requirements.
If your db is relatively small and mysqldump dumps it in seconds or in a few minutes then its ok to just run scheduled backups. For most cases it is sufficient to have a daily backup which runs at a time when your app is mostly idle (at night when you clients are sleeping). You can use a nice tool automysqlbackup for that: it cares about the scheduling and backup rotation, all you need to do is to add it as your cron task and set up its config once.
Setting up a replica is only needed if:
Your backup takes long time (dozens of minutes or hours) to complete so you can not just stop your service for that long.
You can not afford loosing any history in case of main db crash. E.g. if you process financial transactions you may want to ensure that nothing will be lost if master db server dies.
In this cases you may want a replica with backups. Though you must understand that adding replication adds a new layer of problems: replicas may go out of sync, silently crash (and you will not notice that as the master and your app is running fine) etc.
I have been doing some research on best backup procedures for largish (27.678gb) MYSQL database tables.
Currently we are using a program called Rapidsync (which is a offsite backup tool) but it is slow and it locks the tables it's currently backing up therefore causing downtime/slowness of sql.
Our current server is running Windows 2008 r2 with SQL server 2008 (on the same box) also.
Hardware specs for the dedicated server are:
16gb Ram
CPU intel xeon E3-1230 V2 # 3.30GHz
1 TB hard drive
In terms of databases we have 58 in total varying in size in which some need to backed up weekly ideally or even daily.
Through a program we use called Navicat you can tunnel to a database using SSH and copy databases manually, is this a reliable and feasible option if we were to install it on our local machine and copy them across? Or would it be more secure/efficient to use SQL Dump maybe?
I hope I have given all of the necessary info but please do ask if you need to know more.
P.S Only free options at this moment as we are on a tight budget! :)
Thanks in advance
You mentioned SSH, so I suppose the backups can be done also on other server than the database server itself. For Unix, you can use great tool Percona xtrabackup, which is free and supports online (without locking) backup of InnoDB database files and also incremental backups. Maybe it is possible to compile it also on Windows (part of it is written in C, part in Perl).
So you can setup weekly full backup and daily incremental backups. The tool will keep track of which pages of the InnoDB datafile has been changed and will copy only those.
What do I have to consider when backing up a database with millions of entries? Are there any tools (maybe bundled with the MySQL server) that I could use?
Depending on your requirements, there's several options that I have been using myself:
if you don't need hot backups, take down the db server and back up on the file system level, i. e. using tar, rsync or similar.
if you do need the database server to keep running, you can start out with the mysqlhotcopy tool (a perl script), which locks the tables that are being backed up and allows you to select single tables and databases.
if you want the backup to be portable, you might want to use mysqldump, which creates SQL scripts to recreate the data, but which is slower than mysqlhotcopy
if you have a copy of the db at a certain point in time, you could also just keep the binlogs (starting at that point in time) somewhere safe. This can be very easy to do and doesn't interfere with the server's operation, but might not be the fastest to restore, and you have to make sure you don't miss part of the logs.
Methods I haven't tried, but that make sense to me:
if you have a filesystem like ZFS or are running on LVM, it might be a good idea to do a snapshot of the database by doing a filesystem snapshot, because they are very, very quick. Just remember to ensure a consistent state of your db during the whole operation, e. g. by doing FLUSH TABLES WITH READ LOCK (and of course, don't forget UNLOCK TABLES afterwards)
Additionally:
you can use a master-slave setup to replicate your production server to either a different machine or a second instance on the same machine and do any of the above to the replicated copy, leaving your production machine alone. Instead of running continously, you can also fire up the slave on regular intervals, let it read the binlog, and switch it off again.
I think, MySQL cluster and the enterprise licensed version have more tools, but I have never tried them.
Mysqlhotcopy is badly described - it only works if you use MyISAM, and it's not hot.
The problem with mysqldump is the time it takes to restore the backup (but it can be made hot if you have all InnoDB tables, see --single-transaction).
I recommend using a hot backup tool, like what is available in XtraBackup:
http://www.percona.com/docs/wiki/percona-xtrabackup:start
Watch out if using mysqldump on large tables using the MyISAM storage engine; it blocks selects while the dump is running on each table and this can take down busy sites for 5-10 minutes in some cases.
Using InnoDB, by comparison, you get non-blocking backups because of its row-level locking, so this is not such an issue.
If you need to use MyISAM, a common strategy is to replicate to a second MySQL instance and do the mysqldump against the replicated copy instead.
Use the export tab in phpMyAdmin. phpMyAdmin is the free easy to use web interface for doing MySQL administration.
I think mysqldump is the proper way of doing it.