Assume a number of conventional LAMP-style applications which use MySQL as a back-end to record the 'current durable state' for the applications.
I am interested in establishing an 'audit' of transitions at the database level - and storing them as a log. The idea is that - assuming the MySQL database has been 'dumped' at the beginning of the day, it would be possible to 'replay' transactions against the back-up to recover any state during the working day.... A bit like time-machine for MySQL - I guess.
I have found some documentation about "Audit plugins" which look relevant but leaves me with more questions than answers.
http://dev.mysql.com/doc/refman/5.6/en/writing-audit-plugins.html
Essentially, I'd like to establish if it would be feasible to write a MySQL plugin to achieve my goal - such that it would work 'seamlessly' with existing MySQL applications?
The principal detail I'm finding it difficult to ascertain is this: When the audit-plugin is notified of an event, what is the mechanism by which the new data can be established in order to log it? How are data types encoded? How hard would it be to write a tool to 'replay' this audit against a 'full-system-backup' using mysqldump, for example?
Are there any existing examples of such plugins?
You just want MySQL's Point-in-Time (Incremental) Recovery Using the Binary Log:
Point-in-time recovery refers to recovery of data changes made since a given point in time. Typically, this type of recovery is performed after restoring a full backup that brings the server to its state as of the time the backup was made. (The full backup can be made in several ways, such as those listed in Section 7.2, “Database Backup Methods”.) Point-in-time recovery then brings the server up to date incrementally from the time of the full backup to a more recent time
Related
I am working on an application that we need disaster recovery plans. We currently use RDS to host the db and have 2 hourly backups running (we do not use Aurora but have plans to upgrade in future).
If the database somehow got deleted we want to make sure the backup we will be recovering from is current and therefore need some way of telling that.
One way is to save a heartbeat in the db at certain intervals then i can check that against what is expected.
I was wondering if anyone may have any other ways of solving this issue?
Assuming this MySQL database is connected to your web application, you could have a server side thread in your application which periodically does a heartbeat which writes a record to the database. You may create a special heartbeat table, which will store the heartbeats. Then, you can easily examine any backup and know roughly the last time the database were "alive."
I am not an expert in AWS, and there may be another way of doing this which is easier than what I described above.e
I have a application where I need to maintain the audit log operation performed on the collection. I am currently using the MongoDB for storage purpose which work well so far.
Now for audit log I am thinking to use the MySQL database where reasons are-
1. Using the mongo implicit audit filter degrade the performance.
2. Storage will be huge if I store the logs also in the mongoDB which will impact in replication of nodes in cluster.
Conditions to see the logs are not very often in application, so thinking to store logs out of main storage. I am confused to use mongoDB with MySQL, also is this a right choice for future perspective.
Also Is MySQL a good choice to store the audit log, or any other database can help me in storage and conditional query later.
Performance is not guaranteed to go to a completely different database system only for this purpose.
My first attempt for separation would be creating a new database in your current database system and forward to there or even using a normal text file.
Give your feedbacks.
Im researching the best way of logging queries in MySQL database. The log should be used for two things:
Documentation of system activity
Used to recreate the database (in case the database is hacked or otherwise corrupted)
It's possible to log all queries in a MySQL database (like this example)
Question: It is possible to recreate a database on the basis of the log file, or should I use a different approach?
You can use replication logs for this - they store the complete set of operations. You should be able to create a new database from original sources and apply all changes upon it.
You can do complete dumps (i.e. once a week) and archive the replication logs on daily basis.
I'm setting up a SQL Server 2008 server on a production server, which way is the best to backup this data? Should I use replication and then backup that server? Should I just use a simple command-line script and export the data? Which replication method should i use?
The server is going to be pretty loaded so I need an efficent method.
I have access to multiple computers that I can use.
A very simple yet good solution is to run a full backup using sqlcmd (formerly osql) locally, then copy the BAK file over the network to a NAS or other store. It's sub-optimal in terms of network/disk usage, but it's very safe because every backup is independent and given that the process is very simple it is also very robust.
Moreover, this even works in Express editions.
The "best" backup solutions depends upon your recovery criteria.
If you need immediate access to the data in the event of a failure, a three server database mirroring scenario (live, mirror and witness) would seem to fit - although your application may need to be adapted to make use of automatic failover. "Log shipping" may produce similar results (although without automatic failover, or need for a witness).
If, however, there's some wiggle room in the recovery time, regular scheduled backups of the database (e.g., via SQL Agent) and it's transaction logs will allow you to do point-in-time restores. The frequency of backups would be determined by database size, how frequently the data is updated, and how far you are willing to rollback the database in the event of complete failure (unless you can extract a transaction log backup out of a failed server, you can only recover to the latest backup)
If you're looking to simply rollback to known-good states after, say, user error, you can make use of database snapshots as a lightweight "backup" scenario - but these are useless in the event of server failure. They're near instantaneous to create, and only take up room when the data changed - but incur a slight performance overhead.
Of course, these aren't the only backup solutions, nor are they mutually exclusive - just the ones that came to mind.
We are using MySQL version 5.0 and most of the tables are InnoDB. We run replication to a slave server. We are thinking of backing up the MySQL log files on a daily basis.
Questions
Is there any other way of doing an incremental backup without using the log files?
What are the best practices when doing incremental backups?
AFAIK the only way of doing incremental backups is by using the binary-log. You have other options if you want to do full backups (InnoDB hotcopy), but incremental means that you need to log all transactions made.
You need to ask yourself why you're backing up data. Since you have a slave for replication, I assume the backup is primarly for reverting data in case of accidental deletion?
I would probably rotate the logs every 1 hour and take a backup of it. Meaning, restoring would leave the data at most 1 hour old, and you can restore to any point in time since the last full snapshot.
You can dump your schemas regularly with mysqldump, using always the same file name and path for each schema (i.e. replacing the latest one)
Then combine that with any backup tool that supports incremental/delta backup, for example rdiff-backup, duplicity, Duplicati or Areca Backup. An example from duplicity docs:
Because duplicity uses librsync, the incremental archives are space
efficient and only record the parts of files that have changed since
the last backup
That way your first backup would be the compressed copy of the 1st full dump, and the second would contain the compressed differences from the 1st and 2nd dump and so on. You can restore the mysqldump file of any point in time and then restore that file into MySQL.
A lot of time has passed since the last answer, and during this time several solutions and tools have appeared for implementing incremental backups.
Two main ones:
Percona XtraBackup - is an open-source hot backup utility for MySQL -
based servers that doesn’t lock your database during the backup. It
also allows you to create incremental backups. More details here.
It's pretty simple and looks something like this:
xtrabackup --backup --target-dir=/data/backups/inc1 --incremental-basedir=/data/backups/base
mysqlbackup is an utility, that is included in the mysql enterprise
edition. It is a lot like percona xtrabackup. A detailed comparison
can be found here
It has the parameter --incremental which allows you to make incremental backups. More details here
mysqlbackup --defaults-file=/home/dbadmin/my.cnf --incremental --incremental-base=history:last_backup --backup-dir=/home/dbadmin/temp_dir --backup-image=incremental_image1.bi backup-to-image
These two utilities make physical backups (copy database files), but you can still make logical backups of binlog files.
You can either write a script by yourself or use a ready-made script from github:
macournoyer/mysql_s3_backup
Abhishek-S-Patil/mysql-backup-incremental
There are also paid solutions that are, in fact, beautiful wrappers for these tools:
SqlBak
databasethink
What are the best practices when doing incremental backups?
It all depends on your architecture, the amount of data, the maximum allowable downtime interval that is acceptable for you. The maximum allowable data loss interval. Consider these things before setting up backups.
I would like to mention only one good practice, but very important, and which is very often forgotten. Test and run the recovery script regularly on another unrelated server.