I am working on a development copy of a MySQL / InnoDB database that is about 5GB.
I need to restore it pretty frequently for testing alter scripts, and using the mysqldump file is taking quite a while. The file itself is about 900MB and takes around an hour to load in. I've removed data inserts for unimportant tables, and done extended inserts, etc., but it's still pretty slow.
Is there a faster way to do this? I am thinking to just make a copy of the database files from .../mysql/database-name, id_logfile#, and ibdata1, and copy them back in when I need to 'reset' the db, but is this viable with InnoDB? Is the ibdata file for one database? I only see one, even though I have multiple InnoBD db on this box.
Thanks!
I believe you can just copy the file named for the database (with the server daemon down) and all should be well. It seems like something that a little testing on a sample db should answer quickly, no?
I have no idea if it would be faster -- it might be slower, in fact, depending on how extensively your tests alter the data --, but what if you put all your tests in a transaction, and then rolled it back?
Use an LVM snapshot of the disk. Load your data (lengthy process) and take a snapshot (seconds). Then, whenever you need to go back to the snapshot, play LVM games -- again only seconds to re-load and re-clone any sized disk.
You might be able to do something with http://dev.mysql.com/doc/refman/5.1/en/multiple-tablespaces.html
See the ALTER TABLE xxx IMPORT TABLESPACE option especially.
For innodb backup and restore speed you might want to look at mydumper/myloader - much faster than mysqldump due to multi-threaded nature http://vbtechsupport.com/1716/
Related
I read somewhere that it is, so I was wondering.
Don't do that, the .frm/.myd/.myi may be much larger than the actual data and may cause crash (data not consistence) or very hard to transfrom/recover.
Use mysqldump to transfer MySQL database.
It might or might not work, depending on the state of the data.
it should work if you stop the database completely and restart it again after the backup, but there you have a downtime.
A slightly more detailed response at askers request.
Some details of the dangers of a straight file copy:
If the database is live the database might change some of the files
before others so when you copy them you copy them you may get copies
in that are inconsistent with each other. If the database is offline
this method is probably reliable.
Advantages of using the documented methods:
Should work on future version of DBMS
Should work consistently across underlying engines
Always a consistent snapshot like copy
I'm quite new to MySQL and I was wondering: when dumping a mysql database it takes only a few seconds, but when loading it sometimes it takes a few minutes! Is there a reverse of mysqldump to load the database in a few seconds?
Some easy tuning here might help.
Also, there are techniques that can help in specific situations, such as using --disable-keys.
In addition, there is an older post. Be careful of the chosen answer though, the comment said it is dangerous, which is correct, and this tool is now officially deprecated.
In mysql, for storage engines that use file-based storage, you can backup and restore using the files. See this relevant page:
http://dev.mysql.com/doc/refman/5.1/en/backup-methods.html
It's slower when you load it because it has to recreate the indexes. So the short answer is "no". However you can improve it by using --opt option when you dump. This adds some SQL to the dump file that does various things such as disabling the keys until all the data is loaded so it rebuilds indexes all at once.
This offers a nice improvement.
No - writes always take longer than reads - and with a relational database it has to rebuild the indexes too. There are some things you can do to make it go faster though (e.g. use extended inserts, defer index rebuilds)
there is an interesting to restore from mysql dump files (created using mysqldump) ... the technique is explained in detail at
http://www.geeksww.com/tutorials/database_management_systems/mysql/tips_and_tricks/fast_parallel_restore_from_sql_dumps_mysqldump_for_mysql.php
it uses different user accounts to run backups in parallel
I'm trying to build a dev copy of a production MySQL database by loading one of the backups. How long should it take to do this if the uncompressed dump is ~20G?
This command has been running for something like 24h with 10% CPU load and I'm wondering if it's just slow or if it/I am doing something wrong.
mysql -u root -p < it_mysql_dump.sql
BTW it's on a beefy desktop dev machine with plenty of ram, but it might be reading and writing the same HDD. I think I'm using InnoDB.
Restoring MySQL dumps can take a long time. This is because it does really rebuild the entire tables.
Exactly what you need to do to fix it depends on the engine, but in general
I would say, do the following:
Zeroth rule: Only use a 64-bit OS.
Make sure that you have enough physical ram to fit the biggest single table into memory; include any overhead for the OS in this calculation (NB: On operating systems that use 4k pages i.e. all of them, the page tables take up a lot of memory themselves on large-memory systems - don't forget this)
Tune the innodb_buffer_pool such that it is bigger than the largest single table; or if using MyISAM, tune the key_buffer so that it is big enough to hold the indexes of the largest table.
Be patient.
Now, if you are still finding that it is slow having done the above, it may be that your particular database has a very tricky structure to restore.
Personally I've managed to rebuild a server with ~ 2Tb in < 48 hours, but that was a particular case.
Be sure that your development system has production-grade hardware if you intend to load production data into it.
In particular, if you think that you can bulk-load data into tables which don't fit into memory (or at least, mostly into memory), forget it.
If this all seems like too much, remember that you can just use a filesystem or LVM snapshot online with InnoDB, and then just copy the files. With MyISAM it's a bit trickier but can still be done.
Open another terminal, run mysql, and count the rows in some of the tables in your dump (SELECT COUNT(*) FROM table). Compare to the source database. That'll tell you the progress.
I INSERTed about 80GB of data into MySQL over a network in about 14 hours. They were one-insert-per-row dumps (slow) with a good bit of overhead, inserting on a server with fast disks.
24 hours is possible if the hardware is old enough, or your import is competing with something else for disk IO and memory.
I just went through the experience of restoring a 51.8 Gb database from a 36.8 Gb mysqldump file to create an imdb database. For me the restore which was not done over the network but was done from a file on the native machine took a little under 4 hours.
The machine is a Quad Core Server running Windows Server 2008. People have wondered if there is a way to monitor progress. There actually is. You can watch the restore create the database files by going to the Program Data directory and finding the MYSQL subdirectory and then finding the subdirectory with your database name.
The files are gradually built in the directory and you can watch them build up. No small comfort when you have a production issue and you are wondering if the restore job is hung up or just taking a long time.
I just spend a few hours putting together a table full of data, and just decided to drop a WP installation in the same database. I expected WordPress to leave my table as it was and simply add the WP_* tables, but apparently the install wiped my database before installing the wordpress tables.
I have full access to the server root. Is there any way for me to recover this table?
I'm sorry, but that can't be done. Once a table is truncated or dropped everything is gone.
The only way to get it back would be recovering the database files from the disk, but that is not very reliable. To do that, you first have to find out where the MySQL's data files are (see this), then try using a file recovery tool to get them back. Good luck.
Another one who learns to back up the hard way.
If tables were MyISAM, then I agree with NullUserException - you can't recover the tables. Well, the chance is very slim.
If InnoDB - check out my presentation on slideshare - there is a tool to extract records from raw bytes stream - percona data recovery toolkit. There are examples in the presentation, your scenario is pretty much typical.
UPDATE: Data recovery toolkit moved to GitHub
I have 200GB / 400Mrows mysql/innodb database - far beyond what's reasonable as I found out.
One surprising problem is restoring backups. mysqldump generates huge sql files, and they take about a week to import back into a fresh database (attempts at making it faster like bigger/smaller transactions, turning off keys during import etc., network compression etc. failed so far, myisam import seems 2x faster but then there would be no transactions).
What's worse - and I hope to get some help with this - a network connection which transfers >200GB over a time period of a week has a non-trivial chance of breaking, and sql import process cannot be continued in any non-trivial way.
What would be the best way of dealing with it? Right now if I notice a broken connection I manually try to figure out when it ended by checking highest primary key of the last imported table, and then have a perlscript which basically does this:
perl -nle 'BEGIN{open F, "prelude.txt"; #a=<F>; print #a; close F;}; print if $x; $x++ if /INSERT.*last-table-name.*highest-primary-key/'
This really isn't the way to go, so what would be the best way?
Does your MySQL box have enough hard drive space for all the data doubled? Local storage would be best here, but if it's not an option, you could also try some sort of NAS device utilizing iSCSI. It's still happening over the network, but in this case you get more throughput and reliability, because you're only relying on a NAS which has a pretty slim OS and almost never has to be rebooted.
You can't use mysqldump to backup large databases - 200G is feasible but bigger ones it gets worse and worse.
Your best bet is to take a volume snapshot of the database directory and zip that up somehow - that's what we've generally done - or rsync it somewhere else.
If your filesystem or block device does not support snapshots then you're basically in trouble. You can shut the db down to take a backup, but I don't imagine you want to do that.
To restore it, just do the opposite then restart and wait (possibly some time) for innodb recovery to fix things up.
The maatkit mk-parallel-dump and restore tools are a bit better than mysqldump, speed-wise - but I'm not 100% convinced of their correctness
Edit: re-reading the question, I think filesystem snapshot + rsync is probably the best way to go; you can do this without impacting the live system (you'll only need to transfer what changed since the last backup) too much and you can resume the rsync if the connection fails, and it'll continue where it left off.
Do you need everything in the database?
Can you push some of the information to an archive database and add something into your application that would allow people to view records in the archive,
Obviously this depends a lot upon your application and set up, but it may be a solution? Your DB is probably only going to get bigger....