I'm looking for some insight whether or not rsyncing a copy of the data folder from MariaDB well running in docker will provide a usable backup. I'm deploying several containers with mapped folders in a production environment using docker.
I'm thinking of using rsnapshot for nightly backups as it uses hardlinks incrementally and I can specify the number of weekly / daily / monthly copies to keep.For the code and actual files I suspect this will work wonderfully.
For MariaDB I could run mysqldump every night but this would essentially use a new copy of the database each time instead of an incremental one. If I could rsync the data folder and be 100% sure the backup would be fully intact it would be advantageous I presume. Is there any chance this backup method would fail if data was written during the rsync? Would all the files inside of MariaDB change with daily usage (it wouldn't be advantageous if so)?
This is probably a frequent question, but I can't find a really exact match right now.
The answer is NO — you can't use filesystem-level copy tools to back up a MySQL database unless the mysqld process is stopped. In a Docker environment, I would expect the container to stop if the mysqld process stops.
Even if there are no queries running, the InnoDB engine is probably doing writes in the background to flush pages from memory into the tablespace, clean up rolled-back transactions, or finish some deferred index merges.
If you try to use rsync or cp or any other filesystem-level tools to copy InnoDB files, you will only get corrupted files that can't be restored.
Some people use LVM snapshots to get an atomic snapshot of the whole filesystem as of a single instant, and this can be used to get quick backups.
Another useful tool is Percona XtraBackup, which copies the InnoDB tablespace files while it is also copying the InnoDB transaction log continually. Only with both of these in sync can the backup be restored. Read the documentation here: https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html
At my current job, we use Percona XtraBackup to make nightly backups for thousands of MySQL instances. We run Percona Server (not MariaDB) in Docker pods, and Percona XtraBackup runs as another container in the pod. It works very well, and it's free, open-source software.
Related
Percona XtraBackup gets a lot of praise, from what I can see, but I find it incredibly frustrating. I'm using:
root#GR-00258:~# xtrabackup --version
xtrabackup version 2.4.9 based on MySQL server 5.7.13 Linux (x86_64) (revision id: a467167cdd4)
I can create backups of one or more single databases without problems, but there doesn't appear to be any way to restore them. The only way I have found is to restore them as a full backup into an empty /var/lib/mysql, which means mysql won't start up, of course. It seems like a remarkably poor tool for restore - what is the purpose of being able to make a backup of individual databases, if they cannot be restored?
Enough ranting - is there a way to get this to work, or am I just wasting my time? I know, I can use mysqldump into a csv file, but that is not an attractive option, when the databases are ~500GB - ~1TB.
Percona XtraBackup allows you to make a physical backup of a MySQL datadir without blocking access for clients. It's significantly faster than mysqldump to make a backup.
Restoring is also very fast. All you have to do is copy the backup files to a new datadir (after doing the prepare step, which you could do at the time you create the backup).
When restoring a full backup, you can't do that when the destination MySQL Server is running. You must shut down mysqld then copy the files into place, make sure the files have the right ownership and permissions, then start mysqld.
It's not easy to import selected tables or schemas without overwriting. But it is possible:
You must use the --export option when you prepare the backup. Then you can import individual tablespaces to an existing MySQL datadir. But you must do this one tablespace at a time, unfortunately. There's no way to do it in one step for all tables in a schema. You should be able to write a script to do that.
See a complete example of importing tablespaces from a backup here: https://www.percona.com/doc/percona-xtrabackup/8.0/xtrabackup_bin/restoring_individual_tables.html
A MySQL database is used for test purpose, so I want to run backend API tests on this (multiple transactions), simulating a real database in production.
When tests execution are finished, I want to restore the database in the point before that execution, to be able to run more tests with that database snapshot.
I don't have any MySQL version restriction, because it'll be a fresh database server.
What's the most performant way to restore this database?
I'd use Percona XtraBackup to create a physical backup. This can be restored a lot faster than a dump created by mysqldump.
If the database is large enough (over 100GB), even a physical backup won't be fast enough. I'd use LVM snapshots of the filesystem. See https://www.lullabot.com/articles/mysql-backups-using-lvm-snapshots for example.
You have a lot of options:
Create a snapshot with a tool like mysqldump and restore from this snapshot later to reset the database.
Create a volume-level snapshot on a filesystem that supports it like ZFS and restore it later.
Create a virtual machine and snapshot that, later restoring it to the snapshot. Most VM environments like VirtualBox support this feature.
Use a Docker container with an attached volume you can snapshot and restore.
I have done different type of backup a lot many times before, but I did everything in LINUX, because thats what people use normally :P.
This time its a new app and it runs on client system independently and its windows. So xtrabackup/percona of no help now :(.
And I am not i a favor of using binLog for differential/incremental backup as, to me its both risky and time consuming.
Can any of you please help me out with a reliable option by which I can perform incremental backup in a windows system.( I can not purchase a tool for backup for every system our app will be used).
There is a way of running Percona XtraBackup given that you are familiar with that tool. Although Percona don't plan to create a version that is native to windows, you can run Percona Xtra Backup in a Docker container.
In summary, once you have set up Docker and given it the necessary access, you can run Percona XtraBackup from within the container and it will write the backups to a folder in your C drive.
The full information can be found in this blog post: https://www.percona.com/blog/2017/03/20/running-percona-xtrabackup-windows-docker/
I give you that reference rather than repeating how-to in full because if there are any updates to the procedure then that post is likely to be updated before this answer. I hope this helps.
Disclosure: I work for Percona
There are three ways how you can perform incremental backups on Windows:
Binglog backups
You need to copy new binlog files from time to time to your destination place. This is not very difficult, and in it can be achieved with a simple script.
The main drawback: is a longer recovery time.
mysqlbackup
MySQL Enterprise Edition includes a backup utility: mysqlbackup. It allows you to make physical backups of MySQL, as well as incremental backups. This utility is similar to Percona backup. More details are in the official documentation.
The main drawback: is the price.
XtraBackup in a Docker container
XtraBackup can be run under Windows using Docker. Just map /var/opt/mysql to directory with DB files in Windows C:\Program Files\MySQL\MySQL Server 5.6\data. It is a good alternative to mysqlbackup
The main drawback: is difficulties when using Docker on Windows.
More details about the way how MySQL Incremental backups can be performed please find in this blog post.
--
Choose what you like. For a small database, I would choose a binlog backup - unlike the physical backup that XtraBackup and mysqlbackup do. Binlog backup is a logical backup, binlogs are less "capricious" when restoring, and their redundancy can be fixed by compression.
We have one php application and mysql server running on one of our production server.
Mysql server is currently 4GB big with intention to grow up to tens or even up to hundreds of GB.
What am curious to find out is what are the best practices for backup of mysql database in condition that application must be live under any circumstance? What is better, to have mysql replication server on which we will run backup scripts or to run on live server? What is more likely to slow down We have possibility to add additional server(s) if needed. Where do I need to store mysql dumps? Is it suggested to ftp copy mysql backup files to remote server.
What is the best practice to organize web application backup if don't have problem with number of server instances?
MySQL backup methods are documented on MySQL documentation.
The ideal backup solution will be to use MySQL Enterprise Backup. This is a licensed product sold on Oracle store. It is very fast compared to mysqldump.
MySQL Enterprise Backup: A licensed product that performs hot backups
of MySQL databases. It offers the most efficiency and flexibility when
backing up InnoDB tables, but can also back up MyISAM and other kinds
of tables.
If you are looking for a free solution with MySQL community edition, then you can install another replication server and either run mysqldump to take backup or make a raw data backup. During backup on your replication server, your main master database will be running. Since your data is big or will get bigger, it is recommended to backup raw data files. It is basically a process of copying data and log files from disk. Details are explained on MySQL documentation.
For larger databases, where mysqldump would be impractical or
inefficient, you can back up the raw data files instead. Using the raw
data files option also means that you can back up the binary and relay
logs that will enable you to recreate the slave in the event of a
slave failure.
Finally, you should copy backup files to another physical disk on the same to recover from disk failures or to another physical server to easily recover from complete server failures.
Replication is something that protects against hardware errors, for example, a hard disk crashed.
Backup - protects against software errors, for example, due to the human factor, data has been deleted from a table.
It is definitely good practice to combine both of these technologies by running a utility to create a backup on a replica. This not only reduces the load on the product database, but also covers more recovery scenarios.
In case of a hardware error, you can restore the most up-to-date data from the replica, and in cases of data corruption, you can already consider about from the what date to use the backup for recovery. Well, if your both the main server and the replica fail, then the backup will also save you.
What is the best way to make backups?
mysqldump is a good solution for small databases. This is a utility for creating logical backups nad it is included to MySQL Server. At the output, the utility creates a .sql file to recreate the database.
For large databases, it is better to use a physical backup. There are two ways on how to do it.
mysqlbackup is a utility included with MySQL Enterprise Solution. As a result, you get a binary file. Such a backup is created much faster than using mysqldump and is less load on the server.
xtrabackup, from Percona, is a lot like the MySQL Enterprise backup utility, but it's free. A more detailed comparison can be found here.
How often the backups should be made?
The more often you make backups, the better, but you can't make many such backups - since you will run out of space in the backup storage. There are two ways:
Find a compromise between the frequency of backups and the duration of storage.
Use incremental backups. The above utilities support incremental backups, but the management of such backups is more complicated (read more here)
Where the backups should be stored?
Anywhere you prefer, but not in the same place as the MySQL Server. Overall, I think using cloud storage is a good choice. Almost everyone today has a command line interface.
How to automate a backup?
The process of creating regular backups should be automated, and a person should intervene in it only in case of failure. A good backup process should include the following steps:
Creating a backup copy
Compression\Encryption
Uploading to storage
Sending success\fail notification
Removing old backups from the storage (so that it does not overflow)
The simplest script that implements this can be found, for example, here.
Something else?
Yes, the most important thing is not to create a backup and then restore it. Therefore, it is best practice to regularly test the recovery scenarios.
Happy backups!
What is better, to have mysql replication server on which we will run backup scripts or to run on live server
It depends on your db size (and time needed to dump it using mysqldump) and your reliability requirements.
If your db is relatively small and mysqldump dumps it in seconds or in a few minutes then its ok to just run scheduled backups. For most cases it is sufficient to have a daily backup which runs at a time when your app is mostly idle (at night when you clients are sleeping). You can use a nice tool automysqlbackup for that: it cares about the scheduling and backup rotation, all you need to do is to add it as your cron task and set up its config once.
Setting up a replica is only needed if:
Your backup takes long time (dozens of minutes or hours) to complete so you can not just stop your service for that long.
You can not afford loosing any history in case of main db crash. E.g. if you process financial transactions you may want to ensure that nothing will be lost if master db server dies.
In this cases you may want a replica with backups. Though you must understand that adding replication adds a new layer of problems: replicas may go out of sync, silently crash (and you will not notice that as the master and your app is running fine) etc.
The short question:
Is it safe skip ib_logfile* files while creating custom-made backup of MySQL installation with InnoDB engine used?
Custom-made backup is just copy all (or subset of) /var/lib/mysql + /etc/my.cnf to safe place.
As I know ib_logfile* files are recreated by MySQL on start + the files are really huge, so that may not be wise to store them in backup. And yes, assume MySQL is (correctly) stopped before backup.
It's not safe to back up a database in the manner you're describing.
The reason is that you cannot know if some data pages are still in MySQL Server's memory, pending an I/O flush to disk. So copying the files has a good chance of copying garbage, whether or not you include the ib_log files.
Don't do what you're doing. Use mysqlbackup or InnoDB's Hot Backup product.
edit:
Read Farhan Mashraqi's blog article about InnoDB Performance Optimization, in which he says you can delete the ib_log files if you have shut down MySQL Server.
InnoDB log resizing: is not as simple
as changing option and restarting. You
need to shut down MySQL server and
make sure it shuts down normally
(check error log for errors to
confirm). Then move away InnoDB log
files ib_log*. Then start the MySQL
server and check error log files to
see if it successfully created new log
files.
So yes, you should be able to do a filesystem copy of the ibdata files only. You don't need to include the ib_log files, because they should be recreated as MySQL Server starts up. I reiterate, however, that I do not recommend backing up databases with filesystem copy.
Suppose you back up a database today in this manner, using MySQL 5.0. Someday later you need to restore the backup, but by that time you'll be using some future version like MySQL 6.5 for example. Do the ib_data files still use the same internal format in that future version? Will MySQL 6.5 read those files you backed up today? You can't be sure. The backup format is intended to be more portable.