Is it possible to backup the .MYD file only? (and rebuild the .MYI if/when there is a catastrophic failure)
I'd like to backup rather large tables offsite while minimizing bandwidth usage. Data is critical, index files (5G+) are not. The idea is to run regular backups of the .frm and .myd files and rebuild the indexes iff there were a catastrophic failure (i.e. local backups destroyed by fire or stolen).
Repair with .frm and .myd only gives me an error message. Is there an easy workaround?
No. You must also dump the .frm so that the schema is captured. Yes, the .MYI can be recreated. Yeah, "repair with .frm and .MYD" should work; I have never tested it. You could test it by stopping MySQL, copying those two files to a different database, then trying the repair.
MyISAM is not only deprecated, it is gone from the next release (8.0). There is essentially no valid reason for using MyISAM in the future. And many reasons for not using it.
If the .MYI file is bigger than the .MYD file, you may be doing something wrong. A small percentage of tables do have more indexes than data, but is it is somewhat rare. When someone says "I indexed every column", then I know he is a novice and does not understand that:
Indexes on flags (and other low-cardinality columns) will probably never be used.
"Composite" indexes are often useful.
MySQL rarely uses two indexes in the same SELECT.
If you want to minimize bandwidth usage, I can suggest a few other possible strategies:
Use mysqldump, since the output contains only data, no index content. Indexes will be recreated when you restore the dump.
Compress the backup, then transfer the compressed file. It may be 1/10th the size of an uncompressed backup.
Use replication, so only incremental changes to your database are transferred to your other site. Then run backups on the replica.
It's not safe for your data integrity to do "backups" by copying files out of the data directory. You should use a backup tool like mysqldump.
To fully restore a MyIsam table without the .MYI (i.e. solely from the .frm and the .myd files), run:
REPAIR tableName USE_FRM;
To make a fast and compact backup of the structure+data, run:
FLUSH TABLES tableName WITH READ LOCK;
[make a copy of the .frm and .myd files. No need to copy the .myi]
UNLOCK TABLES
Copying the .myd file provides a much faster backup than using mysqldump, for large tables. I ran a quick test and a 6G table took 6 minutes to backup with mysqldump vs a 5 seconds direct copy. The mysqldump file was the combined size of the myd+myi files. The .myd file can/should be compressed (I use 7z).
Direct copy is one of the several backup methods discussed in MySQL's official documentation
Another approach is to create an empty table like the one to be protected.
For the backup:
1) create an empty table like the original
2) backup the original .frm & .MYD
3) include the empty .MYI in the backup set
4) drop the empty table
The recovery steps:
1) Restore the table (the original .frm & .MYD and the empty .MYI restored to same database as the .frm & .MYD)
2) lock the table to prevent its use
3) run myisamchk (I prefer to run from the OS - index regeneration can be parallelized and analyzed for optimizer statistics)
4) flush table table (needed if repairing from the OS)
5) unlock the table
6) analyze the table (if optimizer statistics need to be gathered)
Related
Can you create a copy of a table by simply making a duplicate of .frm file?
I intend to do something like this:
Suppose my database's name is mydb and it has a table called mytab.
Browse to the folder named mydb in the file system and it has a mytab.frm file.
Then copy the contents of mytab.frm into a file called copy.frm
Then i login to mysql and run the following commands:
use mydb; //Selects the database mydb
show tables; //To see the list of tables. I can see the table named copy.
select * from copy; //This throws the error mentioned in the title.
So what am I missing? What files do you copy to take the backup of a database?
I know that table can be copied by a couple of sql statements But i want to learn something knew so I am experimenting around. Thanks! :)
Just throwing this down as an answer:
You can't do what you have tried for InnoDB tables. InnoDB stores all table data in a single file - ibdata. You can modify this with the innodb_file_per_table setting in my.cnf, but it isn't retroactive, it will only apply to new tables. Even if you DO have file per table setup, you still shouldnt try and just copy the data files, because innodb may not have flushed all changes from the ib_logfile's to the ibdata / .ibd file, so you could well end up with corrupt data.
You can do it for MyISAM tables but you shouldn't (and there are also other files that to be copied as well, as the .FRM is only the table definition. The .MYD file contains the data and the .MYI file contains the indexes). Why? because you are entrusting your data to a database, you should be using database tools to duplicate it. The only time you should be touching the data files directly is during data recovery, and only when the server is not running - you dont want to be copying the files as they're being written to.
To duplicate a table, simply do this:
create table new_table as select * from old_table
To backup an entire database, use mysqldump or one of the other available backup tools.
TL;DR
Copying/moving MySQL tables by altering the underlying files is possible in some conditions but it is highly unrecommended.
Always use MySQL commands to do it.
The .frm file contains only the table definition. The data and the indexes are stored in other files and they depend on the storage engine of the table.
Several excerpts from the official documentation:
MyISAM
15.2 The MyISAM Storage Engine
Each MyISAM table is stored on disk in three files. The files have names that begin with the table name and have an extension to indicate the file type. An .frm file stores the table format. The data file has an .MYD (MYData) extension. The index file has an .MYI (MYIndex) extension.
InnoDB
14.1 Introduction to InnoDB
By default, with the innodb_file_per_table setting enabled, each new InnoDB table and its associated indexes are stored in a separate file. When the innodb_file_per_table option is disabled, InnoDB stores all its tables and indexes in the single system tablespace, which may consist of several files (or raw disk partitions).
14.2.15.1 Role of the .frm File for InnoDB Tables
MySQL stores its data dictionary information for tables in .frm files in database directories. Unlike other MySQL storage engines, InnoDB also encodes information about the table in its own internal data dictionary inside the tablespace. When MySQL drops a table or a database, it deletes one or more .frm files as well as the corresponding entries inside the InnoDB data dictionary. You cannot move InnoDB tables between databases simply by moving the .frm files.
14.12 InnoDB Startup Options and System Variables
innodb_file_per_table
When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table in a separate .ibd file, rather than in the system tablespace.
MySQL Glossary
system tablespace
One or more data files (ibdata files) containing the metadata for InnoDB-related objects (the data dictionary), and the storage areas for the undo log, the change buffer, and the doublewrite buffer. Depending on the setting of the innodb_file_per_table, when tables are created, it might also contain table and index data for some or all InnoDB tables. The data and metadata in the system tablespace apply to all the databases in a MySQL instance.
Prior to MySQL 5.6.7, the default was to keep all InnoDB tables and indexes inside the system tablespace, (...) In MySQL 5.6.7 and higher, the default is file-per-table mode, where each table and its associated indexes are stored in a separate .ibd file.
Let's draw some (partial) conclusion
Before anything else you have to stop the MySQL server (to be sure all the data is safely stored into files).
If the table you want to copy uses the MyISAM engine then you need to copy/rename the .frm, .MYD and .MYI files having the same name as the table.
If the table uses the InnoDB engine and at the moment when it was created the innodb_file_per_table setting was ON then you need to copy/rename the .frm and .ibd files having the same name as the table.
If the table uses the InnoDB engine and it was created while the innodb_file_per_table setting was OFF then you cannot copy or move the table data from outside MySQL.
If the table uses the MEMORY table then it's enough to copy the .frm file and restart the server. The table data and indexes are stored in memory, there is no file for them and the source table will be empty after the server restart, so you get an exact copy of an empty table ;-)
But wait, there is more!
MySQL implements several other storage engines that are probably less used than the ones mentioned above. Each of them has its own rules of storing the data in files.
And more
If the server you want to hack this way is part of a replication cluster the changes you do either are ignored (do not propagate to the other servers in the cluster if you change a slave server) or break the replication (the slave servers are required to query and update a table they don't have, if you change the master server).
The conclusion
Even if, in certain conditions, copying or moving a table by changing the underline files is possible, it is strongly not recommended.
The correct (and many times the only) way to copy a table is to use the commands provided by MySQL.
13.1.14 CREATE TABLE Syntax
Use LIKE to create an empty table based on the definition of another table, including any column attributes and indexes defined in the original table:
CREATE TABLE new_tbl LIKE orig_tbl;
The copy is created using the same version of the table storage format as the original table. The SELECT privilege is required on the original table.
You can then use:
INSERT INTO `new_tbl` SELECT * FROM `orig_tbl`
to copy the data.
Another way
An alternative way to copy a table without writing SQL commands is to export the table definition and data using mysqldump, open the export file in a text editor, change the table name in all the places where it appears, save the file and import it into the database using the mysql command line tool (or other MySQL client).
You have to copy 3 files: copy.frm copy.MYD copy.MYI
Make privileges for files, owner and group
chown mysql.mysql copy.*
chmod 660 copy.*
and refresh tables in mysql :
mysql DATABASE
mysql> flush tables;
and voila!
I was looking for ibdata1 file size but got wondered by seeing it just 11MB. There is also another folder by name manu which is same as database name and it is 15GB. ( on my local machine)
I looked into live server and their it is quite opposite!
ibdata1 file is 128 GB and folder having .frm + .MYD +.MYI files size was less.
If I look for statistics (local machine)-
manu DB is 15GB
ibdata1 is 11MB
and along with this, in folder 'manu', every table has 3 files - (just for an example: table NEWS).
NEWS.frm
NEWS.MYD
NEWS.MYI
many times I have dropped all the tables from 'manu' (on my local) and recreated table.
My question is- why the live DB has everything in ibdata (I assume ibdata1 will contain all data what we see in mysql tables) and why on live ibdata1 is very less and files related to them are large in size. Is it not storing all the data in ibdata1 on my local?
What may be the issue.
Actually I wanted to rebuild the DB and set innodb_file_per_table since many deleted table have not released the space and DB size is growing bigger.
This is because your local server uses a different engine for tables than your live server.
When tables are of engine InnoDB their data is stored in ibdata1, when they are stored in .MY* files they are of engine MyISAM.
In a database there can be even a mix of tables with different engines.
The main difference is, that InnoDB is capable of transactions. Which means that statements can get reverted when anything fails, while this is not possible with MyISAM.
A default engine for newly created tables can be specified when creating the database. I guess that happened in your case. You can simply dump your tables and in the created backup script you can replace the engine at the end of each CREATE TABLE statement. Then insert the data again and you're fine.
I Know that this is a bit old question, but an important thing to add is that ibdata stores the data from InnoDB Tables by 2 ways:
1- Without my.cnf (my.ini) innodb_file_per_table enabled, all the data from InnoDB tables are stored into the ibdata file, if somewhat it corrupts, congratulations, you just lose everything!
2- with my.cnf (my.ini) innodb_file_per_table enabled, each table (.frm) will have its own data file (.ibd) and ibdata1 will be used as "cache"/"working file", if you corrupt your mysql you can recover your data with the .ibd files.
Some ways to recover your tables for dump (after the operations the tables become useless) are as follows:
http://www.chriscalender.com/recovering-an-innodb-table-from-only-an-ibd-file/
http://dev.mysql.com/doc/refman/5.0/en/forcing-innodb-recovery.html
http://www.quora.com/Jordan-Ryan/Web-Dev/How-to-Recover-innoDB-MySQL-files-using-MAMP-on-a-Mac
If you lose your data, never, EVER use "innodb_force_recovery" before backing up your files, as this will destroy your ibdata files, even with values 1, 2 and 3, there is a risk, with values 4, 5 and 6, its certain to cause damages as they're a more agressive way to force mysql to read the data for dumping. (details: http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_force_recovery)
Lastly but not less important, to avoid getting a corrupt ibdata file and losing your data, ALWAYS EXPLICITY DECLARE innodb_fast_shutdown=0 in your my.cnf (my.ini) file, as its default value if you don't declare it, is "1", fast shutdown will ignore some security operations and can easily corrupt your ibdata file.
details: http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_fast_shutdown
More InnoDB Parameters:
http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html
Is it possible the create a unique ibdate file for each database made in InnoDB ?
Because if I have multiple database made in InnoDB it will all store it in the ibdata1, ibdata2, etc...
Which make it difficult to just restore 1 database and not all clients who have InnoDB. MyISAM create multiple .frm, MYD and MYI file for each table which make it easier to restore.
How can we make in sort that restore is more easy on InnoDB. Do I have to make a DUMP .sql file or there's other solution ?
Thanks
You can't separate InnoDB data by database, but you can separate it by table, and the tables are stored under the respective subdirectory just like MyISAM tables would be. See http://dev.mysql.com/doc/refman/5.1/en/innodb-multiple-tablespaces.html
But regardless of whether you use a central tablespace file or file-per-table, you should not back up or restore InnoDB databases by moving files around on the filesystem. Even when you're not running queries to write to InnoDB tables, there are background threads that are processing the ib_logfile, undo logs, insert buffers, and other things. Your data is spread in multiple places at any given time, and if you try to move InnoDB files around, you will corrupt them.
Instead, use mysqldump to make logical dumps of InnoDB data safely while MySQL is running.
I have a serious problem in my MySQL tables , once there were InnoDB tables which were IN USE and now are somehow hidden
look at this [pic] *Link removed - the number of tables in heading is 79 and actual counted number is 74.
these tables are those that were IN USE
I don't have any recent backup of my database , so this would game of life and death for me
I checked my VPS, I found them at /etc/lib/mysql/db_name/.
EDIT :
I Searched around internet and I found out that every table should have 3 files related to it.
For example, the table table_users has:
-- table_users.frm
-- table_users.MYD
-- table_users.MYI
and for those hidden table , there are only .frm files and the other two files of a table are missing.
I should change my question to: How to recover a innodb table from a .frm file?
InnoDB does not have those three files
InnoDB data is stored in "ibdata1" for all databases and tables.
the table definition is stored in "tablename.frm"
I would say that your InnoDB file has become corrupted, you may want to have a look at these tools:
https://launchpad.net/percona-innodb-recovery-tool
UPDATED
First of all, about the files:
.frm - table structure;
.myd - table data;
.myi - indexes.
To recover tables, you can try (make backup first):
1) run check table tablename - for all db tables;
2) run repair table tablename - for necessary tables.
UPDATED ONCE AGAIN
Another idea... Try this:
Create a new database to restore and create the tables with same name as .frm files (with the one field - only to create new .frm files);
Stop mysql service and replace the created .frm files with yours;
Start mysql service and check.
I expect correct tables (without data, of course). And sorry, for now I have no PC to check, before suggesting...
actually me too was having the same problem with the missing two files. later i found that when the table's type is innodb then the database folder would have only one associated file.
but you can change the table type to myisam to get all three file for the table.
now as per the backup, you can export the database whenever and wherever you want :)
PHP is GREAT :)
![innodb image][1] INNODB SYSTEM TABLESPACE
INNODB system tablespace is contain in the mysql data directory---
INNODB is system tablespace is divde into two parts
1>.frm
it can describe the table format or you can say it is a table *definition*
2>.ibd
it is contain all system related file and it is also contain data and index and ( InnoDB main table space contain – ibdata1 – and redo logs – ib_logfile*.)
ibdata1 contains your InnoDB database and ib_logfile0 and ib_logfile1 are log files for InnoDB.
If you delete your ibdata1 file, then all your InnoDB tables will be lost.
By default, InnDB uses a shared "tablespace," which is one or more files from a single logical storage area. All InnoDB tables are stored together within the tabespace (from all the databases). By default, InnoDB creates two 5MB log files in the data directory: iblogfile0 and iblogfile1. The information is logged in circular fashion, with old information at the front of the log being overwritten when the log fills up.. Consequently, a larger log allows InnoDB to run longer without having to force changes recorded in the logs to be applied to the tablespace on disk.
When recovering a MyISAM table the only file that is strictly needed is the data file (tablename.MYD) - the rest of the files (the index file tablename.MYI and tablename.frm) can be recreated from the data file using REPAIR TABLE.
Assume I'm using InnoDB (with the "innodb_file_per_table" setting) instead - what is the minimum set of files needed to recover a database table from file?
As long as you have the data file and the log files then InnoDB will be able to recover. See this page (and containing chapter) on MySQL.com for more. InnoDB recover is quite different to MyISAM in that is more "built-in" as it were.
To recover MyISAM table you need frm and MYD file. myisamchk can rebuild the index (MYI file)
For InnoDB - depends on innodb_file_per_table. If it's OFF (default), your data in ibdata1. But you need .frm file too.
If it's ON - you need ibdata1, the respective .ibd and .frm file.