A MySQL database is used for test purpose, so I want to run backend API tests on this (multiple transactions), simulating a real database in production.
When tests execution are finished, I want to restore the database in the point before that execution, to be able to run more tests with that database snapshot.
I don't have any MySQL version restriction, because it'll be a fresh database server.
What's the most performant way to restore this database?
I'd use Percona XtraBackup to create a physical backup. This can be restored a lot faster than a dump created by mysqldump.
If the database is large enough (over 100GB), even a physical backup won't be fast enough. I'd use LVM snapshots of the filesystem. See https://www.lullabot.com/articles/mysql-backups-using-lvm-snapshots for example.
You have a lot of options:
Create a snapshot with a tool like mysqldump and restore from this snapshot later to reset the database.
Create a volume-level snapshot on a filesystem that supports it like ZFS and restore it later.
Create a virtual machine and snapshot that, later restoring it to the snapshot. Most VM environments like VirtualBox support this feature.
Use a Docker container with an attached volume you can snapshot and restore.
Related
I'm looking for some insight whether or not rsyncing a copy of the data folder from MariaDB well running in docker will provide a usable backup. I'm deploying several containers with mapped folders in a production environment using docker.
I'm thinking of using rsnapshot for nightly backups as it uses hardlinks incrementally and I can specify the number of weekly / daily / monthly copies to keep.For the code and actual files I suspect this will work wonderfully.
For MariaDB I could run mysqldump every night but this would essentially use a new copy of the database each time instead of an incremental one. If I could rsync the data folder and be 100% sure the backup would be fully intact it would be advantageous I presume. Is there any chance this backup method would fail if data was written during the rsync? Would all the files inside of MariaDB change with daily usage (it wouldn't be advantageous if so)?
This is probably a frequent question, but I can't find a really exact match right now.
The answer is NO — you can't use filesystem-level copy tools to back up a MySQL database unless the mysqld process is stopped. In a Docker environment, I would expect the container to stop if the mysqld process stops.
Even if there are no queries running, the InnoDB engine is probably doing writes in the background to flush pages from memory into the tablespace, clean up rolled-back transactions, or finish some deferred index merges.
If you try to use rsync or cp or any other filesystem-level tools to copy InnoDB files, you will only get corrupted files that can't be restored.
Some people use LVM snapshots to get an atomic snapshot of the whole filesystem as of a single instant, and this can be used to get quick backups.
Another useful tool is Percona XtraBackup, which copies the InnoDB tablespace files while it is also copying the InnoDB transaction log continually. Only with both of these in sync can the backup be restored. Read the documentation here: https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html
At my current job, we use Percona XtraBackup to make nightly backups for thousands of MySQL instances. We run Percona Server (not MariaDB) in Docker pods, and Percona XtraBackup runs as another container in the pod. It works very well, and it's free, open-source software.
I have done different type of backup a lot many times before, but I did everything in LINUX, because thats what people use normally :P.
This time its a new app and it runs on client system independently and its windows. So xtrabackup/percona of no help now :(.
And I am not i a favor of using binLog for differential/incremental backup as, to me its both risky and time consuming.
Can any of you please help me out with a reliable option by which I can perform incremental backup in a windows system.( I can not purchase a tool for backup for every system our app will be used).
There is a way of running Percona XtraBackup given that you are familiar with that tool. Although Percona don't plan to create a version that is native to windows, you can run Percona Xtra Backup in a Docker container.
In summary, once you have set up Docker and given it the necessary access, you can run Percona XtraBackup from within the container and it will write the backups to a folder in your C drive.
The full information can be found in this blog post: https://www.percona.com/blog/2017/03/20/running-percona-xtrabackup-windows-docker/
I give you that reference rather than repeating how-to in full because if there are any updates to the procedure then that post is likely to be updated before this answer. I hope this helps.
Disclosure: I work for Percona
There are three ways how you can perform incremental backups on Windows:
Binglog backups
You need to copy new binlog files from time to time to your destination place. This is not very difficult, and in it can be achieved with a simple script.
The main drawback: is a longer recovery time.
mysqlbackup
MySQL Enterprise Edition includes a backup utility: mysqlbackup. It allows you to make physical backups of MySQL, as well as incremental backups. This utility is similar to Percona backup. More details are in the official documentation.
The main drawback: is the price.
XtraBackup in a Docker container
XtraBackup can be run under Windows using Docker. Just map /var/opt/mysql to directory with DB files in Windows C:\Program Files\MySQL\MySQL Server 5.6\data. It is a good alternative to mysqlbackup
The main drawback: is difficulties when using Docker on Windows.
More details about the way how MySQL Incremental backups can be performed please find in this blog post.
--
Choose what you like. For a small database, I would choose a binlog backup - unlike the physical backup that XtraBackup and mysqlbackup do. Binlog backup is a logical backup, binlogs are less "capricious" when restoring, and their redundancy can be fixed by compression.
What are the advantages/disadvantages of using database snapshots vs. backing up SQL dumps (from mysqldump)?
I find SQL dump files easy to backup and restore. It seems to be much faster than using database snapshots as well so I am not sure what's the point of using snapshots when they take longer to use.
For additional context, I use MySQL 5.6 and more specifically AWS RDS Aurora (MySQL edition).
Probably the most significant reason people use snapshots is that it can take days to import a large dump file.
Snapshots have several advantages:
It's faster to create a snapshot or restore a snapshot after your database gets very large. You might not yet have a database that is large enough to show this advantage.
You don't need an EC2 instance to run mysqldump on, nor do you need to provision storage for the dump file. The snapshot infrastructure takes care of both of those for you, as you invoke a snapshot in a "serverless" manner via GUI or API.
Snapshot lifecycle management is included in the AWS console. You can auto-expire snapshots, or save them to Glacier, etc. You can do the same with dump files, but you're kind of on your own for managing them.
Dump files have advantages too:
Dumps are in human-readable files. You can even use grep to search for data in a dump file.
Dumps are more-or-less portable between platforms and MySQL versions.
Dumps are a good way to transfer data between RDS and your on-prem data center (although there's a recent announcement that RDS now supports import using Percona XtraBackups).
We have one php application and mysql server running on one of our production server.
Mysql server is currently 4GB big with intention to grow up to tens or even up to hundreds of GB.
What am curious to find out is what are the best practices for backup of mysql database in condition that application must be live under any circumstance? What is better, to have mysql replication server on which we will run backup scripts or to run on live server? What is more likely to slow down We have possibility to add additional server(s) if needed. Where do I need to store mysql dumps? Is it suggested to ftp copy mysql backup files to remote server.
What is the best practice to organize web application backup if don't have problem with number of server instances?
MySQL backup methods are documented on MySQL documentation.
The ideal backup solution will be to use MySQL Enterprise Backup. This is a licensed product sold on Oracle store. It is very fast compared to mysqldump.
MySQL Enterprise Backup: A licensed product that performs hot backups
of MySQL databases. It offers the most efficiency and flexibility when
backing up InnoDB tables, but can also back up MyISAM and other kinds
of tables.
If you are looking for a free solution with MySQL community edition, then you can install another replication server and either run mysqldump to take backup or make a raw data backup. During backup on your replication server, your main master database will be running. Since your data is big or will get bigger, it is recommended to backup raw data files. It is basically a process of copying data and log files from disk. Details are explained on MySQL documentation.
For larger databases, where mysqldump would be impractical or
inefficient, you can back up the raw data files instead. Using the raw
data files option also means that you can back up the binary and relay
logs that will enable you to recreate the slave in the event of a
slave failure.
Finally, you should copy backup files to another physical disk on the same to recover from disk failures or to another physical server to easily recover from complete server failures.
Replication is something that protects against hardware errors, for example, a hard disk crashed.
Backup - protects against software errors, for example, due to the human factor, data has been deleted from a table.
It is definitely good practice to combine both of these technologies by running a utility to create a backup on a replica. This not only reduces the load on the product database, but also covers more recovery scenarios.
In case of a hardware error, you can restore the most up-to-date data from the replica, and in cases of data corruption, you can already consider about from the what date to use the backup for recovery. Well, if your both the main server and the replica fail, then the backup will also save you.
What is the best way to make backups?
mysqldump is a good solution for small databases. This is a utility for creating logical backups nad it is included to MySQL Server. At the output, the utility creates a .sql file to recreate the database.
For large databases, it is better to use a physical backup. There are two ways on how to do it.
mysqlbackup is a utility included with MySQL Enterprise Solution. As a result, you get a binary file. Such a backup is created much faster than using mysqldump and is less load on the server.
xtrabackup, from Percona, is a lot like the MySQL Enterprise backup utility, but it's free. A more detailed comparison can be found here.
How often the backups should be made?
The more often you make backups, the better, but you can't make many such backups - since you will run out of space in the backup storage. There are two ways:
Find a compromise between the frequency of backups and the duration of storage.
Use incremental backups. The above utilities support incremental backups, but the management of such backups is more complicated (read more here)
Where the backups should be stored?
Anywhere you prefer, but not in the same place as the MySQL Server. Overall, I think using cloud storage is a good choice. Almost everyone today has a command line interface.
How to automate a backup?
The process of creating regular backups should be automated, and a person should intervene in it only in case of failure. A good backup process should include the following steps:
Creating a backup copy
Compression\Encryption
Uploading to storage
Sending success\fail notification
Removing old backups from the storage (so that it does not overflow)
The simplest script that implements this can be found, for example, here.
Something else?
Yes, the most important thing is not to create a backup and then restore it. Therefore, it is best practice to regularly test the recovery scenarios.
Happy backups!
What is better, to have mysql replication server on which we will run backup scripts or to run on live server
It depends on your db size (and time needed to dump it using mysqldump) and your reliability requirements.
If your db is relatively small and mysqldump dumps it in seconds or in a few minutes then its ok to just run scheduled backups. For most cases it is sufficient to have a daily backup which runs at a time when your app is mostly idle (at night when you clients are sleeping). You can use a nice tool automysqlbackup for that: it cares about the scheduling and backup rotation, all you need to do is to add it as your cron task and set up its config once.
Setting up a replica is only needed if:
Your backup takes long time (dozens of minutes or hours) to complete so you can not just stop your service for that long.
You can not afford loosing any history in case of main db crash. E.g. if you process financial transactions you may want to ensure that nothing will be lost if master db server dies.
In this cases you may want a replica with backups. Though you must understand that adding replication adds a new layer of problems: replicas may go out of sync, silently crash (and you will not notice that as the master and your app is running fine) etc.
I'm using mysqldump to backup my database. Since the database and webserver are on the same machine, the mysqldump takes all the CPU and the site 'goes down' until the mysqldump finishes.
Is the solution to move database to another machine and do backups on that machine? Are there other alternatives?
It might be a little too much, but I'll suggest to use replication.
There is a master-slave replication with MySQL. This will allow you to have and identical DB (read-only) on another machine at all times and doesn't require your machine to work too hard since it happens all the time.
It's also pretty easy to set up. You can read more about it here:
mysql site description
i use mysql administrator from the old mysql gui tools to create backup from my website to my pc.
~90 mb backup take less than 2 minutes
If you want smooth backups (without interfering the production system) the master-slave replication is a very good way to do it. However you might not want to reserve a server for a backup slave and indeed mysqldump is using a lot of resources.
You can try Percona XtraBackup which is an open source tool. Works on the file system level and much faster than mysqldump. http://www.percona.com/doc/percona-xtrabackup/ You can even try it on your current setup as it doesn't put any locks on tables.