The short question:
Is it safe skip ib_logfile* files while creating custom-made backup of MySQL installation with InnoDB engine used?
Custom-made backup is just copy all (or subset of) /var/lib/mysql + /etc/my.cnf to safe place.
As I know ib_logfile* files are recreated by MySQL on start + the files are really huge, so that may not be wise to store them in backup. And yes, assume MySQL is (correctly) stopped before backup.
It's not safe to back up a database in the manner you're describing.
The reason is that you cannot know if some data pages are still in MySQL Server's memory, pending an I/O flush to disk. So copying the files has a good chance of copying garbage, whether or not you include the ib_log files.
Don't do what you're doing. Use mysqlbackup or InnoDB's Hot Backup product.
edit:
Read Farhan Mashraqi's blog article about InnoDB Performance Optimization, in which he says you can delete the ib_log files if you have shut down MySQL Server.
InnoDB log resizing: is not as simple
as changing option and restarting. You
need to shut down MySQL server and
make sure it shuts down normally
(check error log for errors to
confirm). Then move away InnoDB log
files ib_log*. Then start the MySQL
server and check error log files to
see if it successfully created new log
files.
So yes, you should be able to do a filesystem copy of the ibdata files only. You don't need to include the ib_log files, because they should be recreated as MySQL Server starts up. I reiterate, however, that I do not recommend backing up databases with filesystem copy.
Suppose you back up a database today in this manner, using MySQL 5.0. Someday later you need to restore the backup, but by that time you'll be using some future version like MySQL 6.5 for example. Do the ib_data files still use the same internal format in that future version? Will MySQL 6.5 read those files you backed up today? You can't be sure. The backup format is intended to be more portable.
Related
I'm looking for some insight whether or not rsyncing a copy of the data folder from MariaDB well running in docker will provide a usable backup. I'm deploying several containers with mapped folders in a production environment using docker.
I'm thinking of using rsnapshot for nightly backups as it uses hardlinks incrementally and I can specify the number of weekly / daily / monthly copies to keep.For the code and actual files I suspect this will work wonderfully.
For MariaDB I could run mysqldump every night but this would essentially use a new copy of the database each time instead of an incremental one. If I could rsync the data folder and be 100% sure the backup would be fully intact it would be advantageous I presume. Is there any chance this backup method would fail if data was written during the rsync? Would all the files inside of MariaDB change with daily usage (it wouldn't be advantageous if so)?
This is probably a frequent question, but I can't find a really exact match right now.
The answer is NO — you can't use filesystem-level copy tools to back up a MySQL database unless the mysqld process is stopped. In a Docker environment, I would expect the container to stop if the mysqld process stops.
Even if there are no queries running, the InnoDB engine is probably doing writes in the background to flush pages from memory into the tablespace, clean up rolled-back transactions, or finish some deferred index merges.
If you try to use rsync or cp or any other filesystem-level tools to copy InnoDB files, you will only get corrupted files that can't be restored.
Some people use LVM snapshots to get an atomic snapshot of the whole filesystem as of a single instant, and this can be used to get quick backups.
Another useful tool is Percona XtraBackup, which copies the InnoDB tablespace files while it is also copying the InnoDB transaction log continually. Only with both of these in sync can the backup be restored. Read the documentation here: https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html
At my current job, we use Percona XtraBackup to make nightly backups for thousands of MySQL instances. We run Percona Server (not MariaDB) in Docker pods, and Percona XtraBackup runs as another container in the pod. It works very well, and it's free, open-source software.
I'm trying to create a DataBase for a very little program that just I and 2 more friends use, so the database will be little.
What I want to do is to share this database with these 2 friends, so I thought about storing the data in a OneDrive shared folder.
At the moment, I'm using a txt file as a "database". Which is placed in a shared OneDrive folder, so when my friends execute the program, it can read the data from there and make it sort of real time "online".
The thing is that I can't find the my.ini file in my
C:\Program Files\MySQL\MySQL Server 8.0\
directory, so I can't change the data folder.
Another trouble I have is that I also don't have the Data folder in that directory, instead I've found it in this one:
C:\Program Files\MySQL\MySQL Workbench 8.0 CE
Is it possible to do what I want to do? And how should I proceed?
Do you think I should be using MySQL 5.x version?
Thanks
If you want to share a database with a couple of other users, putting the datadir in OneDrive isn't going to work.
MySQL stores data in the files under the datadir, that's true. They're stored in files called tablespaces which have the .ibd extension.
But as you do INSERT/UPDATE/DELETE operations, data is temporarily stored in memory and in the transaction logs (ib_logfile*. MySQL has to do a delicate coordination between these, to save data in a durable way, while ensuring good performance. It works well, but only if the MySQL Server is the only process writing to the files.
OneDrive is not coordinating with MySQL at all. It will periodically check for files that have changed since it last did a sync. The interval OneDrive checks files is about every 10 minutes, and this is not configurable.
OneDrive may choose to sync the files at a moment after you've done some INSERT/UPDATE/DELETE operations, and the data is modified in RAM, but hasn't yet been updated in the tablespace files on disk.
Once you've committed a change, it must also be safely stored in the transaction log, even if it isn't updated in the tablespace. But if your friends receive the files in this state (transaction log contains changes which aren't present in the tablespace), they can reconstruct the data from your RAM that they didn't get. This is called InnoDB crash recovery. MySQL Server does if automatically if it starts up and finds the transaction log contains changes that aren't in the tablespace. It assumes you had a sudden reboot and lost what was in RAM.
If your friends try to just keep their MySQL Server running continuously, reading a datadir that is simultaneously being updated by their one OneDrive, it'll basically overwrite their files and MySQL will become confused. It only checks to see if it should do crash recovery as MySQL Server starts. So if the files change in unexpected ways while MySQL Server is already running, it'll just conclude that your hard drive has gotten corrupted. It'll probably report a fatal error and shut down MySQL.
Also if your friends try to make changes of their own to the database, their changes would conflict with the updates from OneDrive. Then their attempts to overwrite files would eventually be synchronized in the opposite direction via OneDrive, and would eventually corrupt your database too. This would happen without warning at 10 minute intervals, whenever OneDrive chose to do its file sync.
So I'm afraid OneDrive is not the solution to share your database.
Alternatives that do have a chance of working include:
Hosting a single instance of MySQL Server on a website that you all share, and giving each of you a client that can use the database. A popular free client is phpMyAdmin. That way there'd be only one instance of MySQL Server, one datadir, even though you each would be concurrent clients reading and writing data. This is the simplest solution, most likely to work.
Exporting the data from your MySQL Server instance using mysqldump periodically, and putting the export file on OneDrive, or sending it to your friends through email or any other means. Then they would have to import that data to their MySQL Server manually. This would overwrite any changes they had done to their database, but it wouldn't appear as corruption. If they want to send changes back to you, they could do a similar operation: export their database, put the dump file on OneDrive, then you'd get their dump file and import it to your MySQL Server instance, overwriting any changes you had done locally since the last time you sent your export to your friends.
Use a MySQL add-on for multi-server synchronous replication, such as InnoDB Group Replication or Percona XtraDB Cluster. But this is probably too complex for you to set up if you're a newbie with MySQL.
By some reason these files ibdata1, ib_logfile0, ib_logfile1 were been deleted from hard drive, mysql server was restarted and of course one of the database stoped with error "db.table does not exist!". Is it possible to restore somehow these files, if there are only *.frm, db.opt files?
Thanks.
Take disk image as soon as possible or mount MySQL partition read-only(if it was not on /).
Then you need to compile data recovery toolkit(install dependencies if "make" fails):
make
Then scan the disk image with page_parser. It will find InnoDB pages and sort them per page type, per index_id.
./page_parser -f /path/to/disk.img
Then you need table structures (either take them from old backup or restore from .frm). You need the structure to compile constraints_parser - a tool that fetches records from InnoDB page(s).
./constraints_parser -5f pages-XXXXXX/FIL_PAGE_INDEX/0-x/
Save output in some file. It will also generate LOAD DATA to stderr to upload the dump back to MySQL.
Check the documentation on Percona's website for more details.
From my experience, 70-80%% of data should be recoverable after this kind of accident.
Good day to you.
I used my MySQL server with "innodb_file_per_table" option, and now server is crashed. I want to recover this server using this way:
Uninstall old MySQL
Install new MySQL
Add "innodb_file_per_table" in MySQL configuration
Copy databases folders (only my, not mysql) from old MySQL/data to new MySQL/data
In every folder I have two files, .frm and .ibd, and looks like this files have all data from my databases tables.
But, after copying, tables in this databases didn't work: when I try to open some table, I get error: Table xxx doesn't exist in engine.
I tried REPAIR command, but it isn't helpful.
If you know how to finish my way of repair — please help.
...I know that I need to copy ibdata1 also, but it looks so dead for recovery, so I try way that I try.
REPAIR command won't help with InnoDB.
If you're lucky enough the best you can do is:
1. Start MySQL with innodb_force_recovery=4 (try 5, 6 values if InnoDB fails to start). Make sure innodb_purge_threads=0.
2. Dump the database with mysqldump tool. Yes, it may be slow, but there is no other choice.
3. Create new empty InnoDB table space and reload the dump.
If MySQL fails to start with innodb_force_recovery=6, then recovery from backups is the only option. Well, you can fetch records from *.ibd files, but this is tedious job - google data recovery tool from percona
UPDATE: Data recovery toolkit moved to GitHub
You need to copy everything, not only the data folder
For example, without the ibdata file, mysql don't know where the tables are stored.
https://serverfault.com/questions/487159/what-is-the-ibdata1-file-in-my-var-lib-mysql-directory
I have a script that needs to add data to a folder created in the mysql data folder and then in some way force the mysql server to reload the data so the database reliably shows up. I am currently using a system call to the /etc/init.d/mysql script to restart the server but this is quite slow, I couldn't find anyhting in mysql that would reload the database through a query of some kind.. I was just wondering if there was a way to do this that was slightly quicker?
This depends on the storage engine you're using.
MyISAM tables will be reloaded automatically as soon as the table is next touched by MySQL (obviously, you need to make sure the .MYI, .MYD and .frm files are consistent).
InnoDB is a bit more complex, and depends on whether it's a traditional configuration with all tables in the same tablespace, or if you're using innodb_file_per_table. With the former, you pretty much have to restart the server. With file-per-table, you can copy individual .ibd files, but then you have to use an ALTER TABLE statement to import that tablespace.
See here for information on working with individual .ibd files without restarting the server.
This page is also useful for seeing how to move InnoDB data from place to place.
What do you mean by 'quite slow'?
You could do a dump/load of the new db in a one named orginal_db_name_temp, rename the original DB, and then rename orginal_db_name_temp to orginal_db_name - then you will have your new DB in place.
However, I don't know which method is faster by your means - because a dump&load roundtrip will be probably slower than a file-copy. But renaming the DBs will probably be faster than restarting the server