We want to run BI analytics on a (copy of a) mysql database.
Currently we get a full mysql dump file daily, but it takes up to 16 hours to import this file into our "BI" server. The dump file itself is about 9GB big, resulting in 360 tables with a total of 20GB (on disk); the biggest table file (.ibd) is around 6GB.
I am no mysql expert, but I think that the import takes so much time since the dump file only imports data, and the database needs to re-index everything from scratch...( as a side question: Any thoughts maybe on how to improve the import?) We have been looking at having a separate SSD for the datafiles, added some more RAM/CPU to the server, but that does not really improve the import speed...
Is there a way to have some sort of "snapshot" of the source database, so that it can be copied over as-is instead of importing?
I think about a .zip file with all the .ibd files (and other necessary files)
Yes, importing a large mysqldump is notoriously slow. The dump file contains the definitions of indexes, but not the index storage itself. So the indexes must be rebuilt every time you import the dump file.
At my company we use Percona XtraBackup.
It's a physical backup tool, meaning the result is not an SQL dump file that must be imported. It just makes a copy of the .ibd files and the iblog files to reconcile transactions. The .ibd files contain both rows of data and indexes.
We use this backup & restore solution to clone databases up to 100x as large as yours.
Percona XtraBackup is free and open-source.
There are a few caveats:
Doesn't work on Windows last I checked (I haven't checked in several years, because I don't use Windows).
You can backup without interrupting service on the source instance, but to "restore" you need to shut down your local MySQL Server, copy the backup into the datadir, and restart the MySQL Server.
It's nearly impossible to import just one database to an instance. In other words, backup and restore is for the full instance, with all tables and schemas. This will overwrite anything else you have on your local instance. Whereas mysqldump is more flexible because you can dump & import just one table or just one schema, and you can import to a running MySQL Server instance without stopping it.
It's worth mentioning that if you don't use a proper backup tool, you should not try to make a zip archive of the MySQL data directory of a running MySQL Server. You're almost certain to get a non-consistent copy of the data files, meaning they will be corrupt and not restorable.
Related
I have been working in a project which involves Mysql and PHP. While creating package for testing i have exported mysql database as sql file from MySql Workbench and imported into linux machine's mysql server with
mysql>source mydatabase.sql;
is this the correct way to export and import mysql database?. And also the database file contains creation of schema, inserting data, and creating indexes scripts. Importing takes long time with this file. My immediate manager suggests me to export without indices and import database, then execute index creation scripts. Is this is the correct way? Does indices takes long time while importing database?
Thanks in advance!
Percona Server comes with a modified mysqldump tool so you can choose the --innodb-optimize-keys option. This exports the tables, data, and index definitions like stock mysqldump, but during import it defers creation of secondary indexes until after the data has been loaded into the table. This takes advantage of InnoDB's fast index creation feature.
Another option is to use a physical backup tool, like Percona XtraBackup (as user #Up_One) mentioned. This means that the full data files are backed up, including the indexes. On restore, nothing needs to be rebuilt, because the indexes are already populated. There are pros and cons to using physical backup tools too, but restore time is a big advantage.
Export depends on the MySQL version/vendor you are running !
MySQL(older versions) - uses mysqldump
MySQL(percona) - uses XtraBackup
MySQL(EE - 5.6) - uses MySQL Enterprise Backup which is more complex
and much faster.
Fastest way is to avoid mysqldump.
I like Percona more then the others !
Another way but is more advanced i think is to create the DB on the destination server, and copy the database files directly. Assuming user has access to the destination server's mysql directory:
As root user:
[root#server-A]# /etc/init.d/mysqld stop
[root#server-A]# cd /var/lib/mysql/[databasename]
[root#server-A]# scp * user#otherhost:/var/lib/mysql/[databasename]
[root#server-A]# /etc/init.d/mysqld start
Important things here are: Stop mysqld on both servers before copying DB files, make sure the file ownership and permissions are correct on the destination before starting mysqld on the destination server.
[root#server-B]# chown mysql:mysql /var/lib/mysql/[databasename]/*
[root#server-B]# chmod 660 /var/lib/mysql/[databasename]/*
[root#server-B]# /etc/init.d/mysqld start
With time being your priority here, the use of compression will depend on whether the time lost waiting for compression/decompression (with something like gzip) will be greater than the time wasted transmitting uncompressed data; that is, the speed of your connection.
I prefer to use mysqldump! (Check this: http://dev.mysql.com/doc/refman/5.0/en/mysqldump.html)
By some reason these files ibdata1, ib_logfile0, ib_logfile1 were been deleted from hard drive, mysql server was restarted and of course one of the database stoped with error "db.table does not exist!". Is it possible to restore somehow these files, if there are only *.frm, db.opt files?
Thanks.
Take disk image as soon as possible or mount MySQL partition read-only(if it was not on /).
Then you need to compile data recovery toolkit(install dependencies if "make" fails):
make
Then scan the disk image with page_parser. It will find InnoDB pages and sort them per page type, per index_id.
./page_parser -f /path/to/disk.img
Then you need table structures (either take them from old backup or restore from .frm). You need the structure to compile constraints_parser - a tool that fetches records from InnoDB page(s).
./constraints_parser -5f pages-XXXXXX/FIL_PAGE_INDEX/0-x/
Save output in some file. It will also generate LOAD DATA to stderr to upload the dump back to MySQL.
Check the documentation on Percona's website for more details.
From my experience, 70-80%% of data should be recoverable after this kind of accident.
We have a very large database that we need to occasionally replicate on our dev+staging machines.
At the moment we use mysqldump and then import the database script using "mysql -u xx -p dbname < dumpscript.sql"
This would be fine if it didn't take 2 days to complete!
Is it possible to simply copy the entire database as a file from one server to another and skip the whole export/import nonsense?
Cheers
there are couple of solutions:
have a separate replication slave you can stop at any time and take the file-level backup
if you use the innodb engine - you can take file system level snapshot [eg with lvm] and then copy the files over to your test environment
if you have plenty of tables/databases - you can paralleled the dumping and restoring process to speed things up.
I have many restrictions on where I can run scripts, access sources and targets, and have enough space to prepare the data for the task.
I get my zipped database dump from the hosting provider.
I split the unzipped commands so INSERT INTO lines get put into one file, and all the others go into a second one.
Then I create the database structures from the second one.
I convert the INSERT INTO statements to table related CSV files.
Finally, I upload the csv files in parallel (up to 50 tables concurrently) and this way a 130GB text file dump is cloned in 3 hours, instead of the 17 hours it'd take when using the statement by statement method.
In the 3 hours, I include:
the copy over (10 minutes),
sanity check (10 minutes) and
filtering of logs (10 minutes), as the log entries need to be from the latest academic year only.
The remote zipped file is between 7GB to 13GB passed over a 40MBps line.
The upload is to a remote server via a 40MBps line.
If your mysql server is local, the speed of uploading can be faster.
I utilise scp, gzip, zgrep, sed, awk, ps, mysqlimport, mysql and some other utilities to speed up decompression and filtering (pv, rg, pigz) if available.
If I had direct access to the database server, an LVM with folder level snapshot abilities would be the preferred solution, giving you speeds restricted only by the copy speed of the media.
I have a script that needs to add data to a folder created in the mysql data folder and then in some way force the mysql server to reload the data so the database reliably shows up. I am currently using a system call to the /etc/init.d/mysql script to restart the server but this is quite slow, I couldn't find anyhting in mysql that would reload the database through a query of some kind.. I was just wondering if there was a way to do this that was slightly quicker?
This depends on the storage engine you're using.
MyISAM tables will be reloaded automatically as soon as the table is next touched by MySQL (obviously, you need to make sure the .MYI, .MYD and .frm files are consistent).
InnoDB is a bit more complex, and depends on whether it's a traditional configuration with all tables in the same tablespace, or if you're using innodb_file_per_table. With the former, you pretty much have to restart the server. With file-per-table, you can copy individual .ibd files, but then you have to use an ALTER TABLE statement to import that tablespace.
See here for information on working with individual .ibd files without restarting the server.
This page is also useful for seeing how to move InnoDB data from place to place.
What do you mean by 'quite slow'?
You could do a dump/load of the new db in a one named orginal_db_name_temp, rename the original DB, and then rename orginal_db_name_temp to orginal_db_name - then you will have your new DB in place.
However, I don't know which method is faster by your means - because a dump&load roundtrip will be probably slower than a file-copy. But renaming the DBs will probably be faster than restarting the server
I am having to restore a mysql database from .myd and .frm files. And Ii have no idea where to begin, I tried just copying them over while preserving file permisssions but to no avail what other steps do i need to take?
I have a feeling its got something to do with the ib_logfile0, ib_logfile1 and ib_data files. But dont know what to do.
You shouldn't be doing backups by dealing with the raw MySQL files. It's trivially easy to end up with a corrupted database that way. Consider what happens when you start copying the files to your backup medium - the copy will take a non-zero amount of time, during which the database could potentially write new data in various places, including the parts you've already copied. Now you're copying a modified file, a mix of old data and new data. This modified copy is almost guaranteed to be corrupted.
You should be using mysqldump or mysqlhotcopy instead, which guarantee a consistent backup.
However, on the chance that your database is relatively quiet and you managed to get a good clean backup copy, the files you need to restore depend on the type of tables you're restoring. InnoDB stores all of its data in the ib* files, regardless of database/table name. MyISAM uses files in directories named according to the database/table names.
After copying the backup copies into the correct locations, you'll have to restart MySQL, as it will still be accessing the original copies of the files.