How can I restore a partitioned mysql InnoDB table from just the .ibd files of the form TableName#P#pname.ibd ? Chris Calendar's article here
http://www.chriscalender.com/?p=28
works for non-partitioned tables with a single .ibd file, but the "discard" and "import" steps result in a "storage engine doesn't have this option" error for partitioned tables.
Wes Smith in one of the comments at the link above suggests a manual procedure to import one partition at a time, but that did not work for me. If I try to follow his approach by creating a non-partitioned table, moving the first .ibd file renamed as TableName.ibd, and doing an import, the import succeeds, but there are zero rows in the table. The subsequent suggested step of "add back in the partition", which I tried as "alter table TableName partition by range ... (partition pname1 values less than (...) ENGINE=InnoDB)"shockingly replaces the TableName.ibd file (corresponding to the first partition) with a fresh TableName#P#pname1.ibd of a trivial size. I lost one partition's worth of data trying this. I have about 150 partitions to recover.
Any advice on how to recover the data from the .ibd files? Thanks.
So, it turns out that the approach suggested in the comment at the link in the original post does work after all. The deletion of the partition I mentioned in the "alter table..." step above was legitimate as it had been intentionally deleted in the original table before the loss of the ibdata and log files (but mysql leaves the .ibd files undeleted anyway). The recovery process is painfully slow, but I have managed to script it up and hope to let it run to completion over the next several days.
Here are some tips if you find yourself in a similar situation with many partitions to recover. Suppose you had a table TableName with 100 partitions. Typically, the 100 partitions would have consecutive innodb IDs if they were created via a "create table" statement, but in general this may not be the case as partitions may have been added after creation. So, here are the steps:
1) Find out the innodb ID corresponding to each partition using Calendar's method in the blog above as follows. Note that this step does not need a restart of the mysql server. Just create a table like TableName without any partitions and discard its tablespace. Then move the first partition's .ibd file (named something like TableName#P#pname1.ibd) to the database directory as TableName.ibd and try to import it. Look into the mysql error.log (typically /var/log/mysql/error.log) to see what that partition's innodb ID is. Repeat this step for each partition (or as needed) until you have the 2-tuples (innodb_ID_i, partition_boundary_i) for all partitions in a file.
2) Start with an empty innodb state (stop the server, delete ibdata and ib_logfile*, restart server). For each innodb_ID entry in the file in step 1 above, create a table TableName_i like TableName. E.g., if the 100 partitions correspond to the IDs 321, 322,..., 370, 415, 416,...464 (two blocks of 50 contiguous IDs each), then write a script to create 320 dummy tables, 50 tables like TableName, 45 dummy tables, and 50 tables like TableName.
3)
For each TableName_i table created above,
--do
(i) rename table TableName_i to TableName
(ii) alter table TableName discard tablespace // important to do this step before the next one
(iii) mv TableName#P#pname_i.ibd TableName.ibd // with the appropriate directory prefixes
(iv) alter table TableName import tablespace
(v) alter table partition by range (partition_field) (partition pname_i values less than (partition_boundary_i)) // This is the most and only time consuming step
(vi) rename table TableName to TableName_i // or some other name or just dump it to a file
--repeat
Note that all of the above steps are scriptable and do not require restarting the server at any point except at the beginning of step 2 to start with an empty innodb state. Be careful to introduce checks for the success of each sub-step in step 3 before moving to the next, otherwise successive steps may fail and/or .ibd files may get overwritten. If feasible, use a copy in step 3(iii) instead of mv.
A final note: There might be a slightly easier alternative using percona's recovery toolkit using hex editing, but this did not work for my case of partitioned tables. I ran into the same, seemingly unresolved issue, with partitioned tables as noted in one of the comments at http://www.mysqlperformanceblog.com/2011/05/13/connecting-orphaned-ibd-files/ . Your mileage may vary though. If there were a way to avoid recreating partitions (like in step 3(v) above), that would be real nice and quick, but I am not sure if there is one.
Related
I was running an optimize on a 70GB+ data table (after adding a bit column), and the database just crashed.
When browsing the data folder, I have a [table].ibd file and a #sql-*.ibd orphan file.
The only way I could restart mysql daemon was to start it with innodb_recovery_mode = 6.
But when reading the table, the columns have values that do not make any since.
One of the columns that is of type BIGINT just repeats the same number for rows on end...
I tried do DISCARD TABLESPACE and the IMPORT it again - it crashes the service while doing it.
InnoDB warns about corrupted table and also corrupted indexes.
I'm looking for help to try and recover the data, since it is very important.
Is it possible to read the .ibd file and recover the data?
I also used "undrop-for-innodb" to try to recover the data, but the values in the rows make no sense.
Thank you in advance.
My partition schemes looks something like:
ALTER TABLE my_table
PARTITION BY RANGE (integer_field) (
PARTITION p0 VALUES LESS THAN (100) DATA DIRECTORY = '/my_location/partitions/p0' ,
PARTITION p1 VALUES LESS THAN (200) DATA DIRECTORY = '/my_location/partitions/p1' ,
PARTITION p_other VALUES LESS THAN (MAXVALUE) DATA DIRECTORY = '/my_location/partitions/p_other'
);
As expected, data is being stored properly into the partitions, and into the proper directory.
Problem:
Now when I remove/move the directory from the location, like mv /my_location/partitions/p0 /some_other_location/ , the directory gets moved successfully, but data is still query-able from MySQL shell, even after restarting the shell.
My Solution:
To get it working as expected, after moving the directory including the .ibd file, I had to drop the partition explicitly:
ALTER TABLE my_table DROP PARTITION p0;
This removed the partition from the scheme as required and also cleared the data, verified it by again querying the same data.
Assumption / Understanding:
I think that MySQL is caching the data, not sure where and why exactly, which makes it query-able even after partition directory is moved away. Definitely cache is not at connection level as I closed and reopned the shell.
Question:
I expected the data to disappear as soon as the directory p0 was moved away. Is it really necessary to run the drop partition statement each time the directory is moved?
Constraints:
It is sure that p0 directory is moved away only when the p0 partition is no more used. So there will not be any more data required to enter into existing p0 partition
MySQL: 8.0.19
Windows or not?
Did you restart mysqld?
mv on Linux (and close relatives) is really a "rename". And, assuming the target is on the same filesystem, that rename can even involve a different directory.
When a program (eg, mysqld) has a file (eg, the table partition) "open" mysqld, it has control over it -- even if you run off to a shell and rm the file!.
I suspect that when you restart mysqld (for any reason, including reboot), the "data directory" will become messed up.
Aside from the filesystem's partitioning, you must tell MySQL when you "archive" a partition. Read up on "transportable tablespaces" for the version you are using. Here's a writeup for 5.6; 5.7 has some improvements.
I don't see the advantage of using a filesystem partition. With "transportable tablespaces", you can disconnect a MySQL partition from a PARTITIONed table. This turns the partition into a table. Then that table can be deleted, renamed, copied, etc, without impacting the partitioned table. Search for "transportable" in http://mysql.rjweb.org/doc.php/partitionmaint ; there are some links.
As pointed by #Rick James and #Solarflare, moving around the ibd file while it is still in Open state for MySQL does behave in weird way, and tablespaces gets messed up. Following their guidance, and the MySQL Docs, here is the final approach that successfully worked for me (Possibly, the right way to do it):
Lock the table
FLUSH TABLES my_table FOR EXPORT;
This prevents any update/write operation to the desired table and makes the ibd file safe to copy. Additionally, this creates a .cfg file. These steps are very well explained in MySQL Docs
Once the table is "Locked", copy the .ibd file to the desired location for archival. PS: Do not move / delete the source file yet.
cp -r /my_location/partitions/p0 /some_other_location/
Unlock tables to be able to alter the partition
UNLOCK TABLES;
Drop the required partition safely. This also informs the tablespaces.
ALTER TABLE my_table DROP PARTITION p0;
Note that this statement leads to removal of partition, plus the data corresponding to that partition.
I need your advice. Need to reclaim disk space on live server with minimum downtime. We are using:
mysql -- 5.5
innodb table per file -- on
Was a huge table(70% insers/30% deletes -- it means that sometimes we delete rows from this table), this table was dropped with "drop table" command, as we expect, mysql didn't release disk space to OS, but now we need to release free space. OPTIMIZE command is possible way -- but now we don't 100% sure if we will have enough free space on disk to do this command, because previous table was huge. But in mysql documentation written:
You can run OPTIMIZE TABLE to compact or recreate a file-per-table tablespace. When you run an OPTIMIZE TABLE, InnoDB creates a new .ibd file with a temporary name, using only the space required to store actual data. When the optimization is complete, InnoDB removes the old .ibd file and replaces it with the new one.
I highlighted confusing moment in documentation, does it means, that "OPTIMIZE dropped_table;" will use only space for data that really needed, for this situation there is no actual data if we drop table early ?
If the table was created when innodb_file_per_table was OFF, then the table was written to ibdata1, and the space is not returned to the OS by DROP TABLE.
If it had been ON, then there would be a .ibd file. Did you see that file before the DROP? Does it still exist? (It should not still exist.)
When DROP removes the .ibd file, there is a slight lag (depending on the OS) while waiting for the data to be actually freed up by the OS. du and/or df reflects this lag.
OPTIMIZE TABLE will copy the table to tmpdir (I think), then drop the old table and play RENAME games. This requires extra space -- enough for a compete copy of the table. Assuming you really have a .ibd file, it will create another .ibd file (with a temp name) and shuffle files. If tmpdir points to a different "filesystem", the shuffle will involve a copy, not just a move.
If I have not explained things adequately, please provide more details -- actual commands you are proposing, size of table in question, whether the .ibd exists, etc.
Can you create a copy of a table by simply making a duplicate of .frm file?
I intend to do something like this:
Suppose my database's name is mydb and it has a table called mytab.
Browse to the folder named mydb in the file system and it has a mytab.frm file.
Then copy the contents of mytab.frm into a file called copy.frm
Then i login to mysql and run the following commands:
use mydb; //Selects the database mydb
show tables; //To see the list of tables. I can see the table named copy.
select * from copy; //This throws the error mentioned in the title.
So what am I missing? What files do you copy to take the backup of a database?
I know that table can be copied by a couple of sql statements But i want to learn something knew so I am experimenting around. Thanks! :)
Just throwing this down as an answer:
You can't do what you have tried for InnoDB tables. InnoDB stores all table data in a single file - ibdata. You can modify this with the innodb_file_per_table setting in my.cnf, but it isn't retroactive, it will only apply to new tables. Even if you DO have file per table setup, you still shouldnt try and just copy the data files, because innodb may not have flushed all changes from the ib_logfile's to the ibdata / .ibd file, so you could well end up with corrupt data.
You can do it for MyISAM tables but you shouldn't (and there are also other files that to be copied as well, as the .FRM is only the table definition. The .MYD file contains the data and the .MYI file contains the indexes). Why? because you are entrusting your data to a database, you should be using database tools to duplicate it. The only time you should be touching the data files directly is during data recovery, and only when the server is not running - you dont want to be copying the files as they're being written to.
To duplicate a table, simply do this:
create table new_table as select * from old_table
To backup an entire database, use mysqldump or one of the other available backup tools.
TL;DR
Copying/moving MySQL tables by altering the underlying files is possible in some conditions but it is highly unrecommended.
Always use MySQL commands to do it.
The .frm file contains only the table definition. The data and the indexes are stored in other files and they depend on the storage engine of the table.
Several excerpts from the official documentation:
MyISAM
15.2 The MyISAM Storage Engine
Each MyISAM table is stored on disk in three files. The files have names that begin with the table name and have an extension to indicate the file type. An .frm file stores the table format. The data file has an .MYD (MYData) extension. The index file has an .MYI (MYIndex) extension.
InnoDB
14.1 Introduction to InnoDB
By default, with the innodb_file_per_table setting enabled, each new InnoDB table and its associated indexes are stored in a separate file. When the innodb_file_per_table option is disabled, InnoDB stores all its tables and indexes in the single system tablespace, which may consist of several files (or raw disk partitions).
14.2.15.1 Role of the .frm File for InnoDB Tables
MySQL stores its data dictionary information for tables in .frm files in database directories. Unlike other MySQL storage engines, InnoDB also encodes information about the table in its own internal data dictionary inside the tablespace. When MySQL drops a table or a database, it deletes one or more .frm files as well as the corresponding entries inside the InnoDB data dictionary. You cannot move InnoDB tables between databases simply by moving the .frm files.
14.12 InnoDB Startup Options and System Variables
innodb_file_per_table
When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table in a separate .ibd file, rather than in the system tablespace.
MySQL Glossary
system tablespace
One or more data files (ibdata files) containing the metadata for InnoDB-related objects (the data dictionary), and the storage areas for the undo log, the change buffer, and the doublewrite buffer. Depending on the setting of the innodb_file_per_table, when tables are created, it might also contain table and index data for some or all InnoDB tables. The data and metadata in the system tablespace apply to all the databases in a MySQL instance.
Prior to MySQL 5.6.7, the default was to keep all InnoDB tables and indexes inside the system tablespace, (...) In MySQL 5.6.7 and higher, the default is file-per-table mode, where each table and its associated indexes are stored in a separate .ibd file.
Let's draw some (partial) conclusion
Before anything else you have to stop the MySQL server (to be sure all the data is safely stored into files).
If the table you want to copy uses the MyISAM engine then you need to copy/rename the .frm, .MYD and .MYI files having the same name as the table.
If the table uses the InnoDB engine and at the moment when it was created the innodb_file_per_table setting was ON then you need to copy/rename the .frm and .ibd files having the same name as the table.
If the table uses the InnoDB engine and it was created while the innodb_file_per_table setting was OFF then you cannot copy or move the table data from outside MySQL.
If the table uses the MEMORY table then it's enough to copy the .frm file and restart the server. The table data and indexes are stored in memory, there is no file for them and the source table will be empty after the server restart, so you get an exact copy of an empty table ;-)
But wait, there is more!
MySQL implements several other storage engines that are probably less used than the ones mentioned above. Each of them has its own rules of storing the data in files.
And more
If the server you want to hack this way is part of a replication cluster the changes you do either are ignored (do not propagate to the other servers in the cluster if you change a slave server) or break the replication (the slave servers are required to query and update a table they don't have, if you change the master server).
The conclusion
Even if, in certain conditions, copying or moving a table by changing the underline files is possible, it is strongly not recommended.
The correct (and many times the only) way to copy a table is to use the commands provided by MySQL.
13.1.14 CREATE TABLE Syntax
Use LIKE to create an empty table based on the definition of another table, including any column attributes and indexes defined in the original table:
CREATE TABLE new_tbl LIKE orig_tbl;
The copy is created using the same version of the table storage format as the original table. The SELECT privilege is required on the original table.
You can then use:
INSERT INTO `new_tbl` SELECT * FROM `orig_tbl`
to copy the data.
Another way
An alternative way to copy a table without writing SQL commands is to export the table definition and data using mysqldump, open the export file in a text editor, change the table name in all the places where it appears, save the file and import it into the database using the mysql command line tool (or other MySQL client).
You have to copy 3 files: copy.frm copy.MYD copy.MYI
Make privileges for files, owner and group
chown mysql.mysql copy.*
chmod 660 copy.*
and refresh tables in mysql :
mysql DATABASE
mysql> flush tables;
and voila!
I have a serious problem in my MySQL tables , once there were InnoDB tables which were IN USE and now are somehow hidden
look at this [pic] *Link removed - the number of tables in heading is 79 and actual counted number is 74.
these tables are those that were IN USE
I don't have any recent backup of my database , so this would game of life and death for me
I checked my VPS, I found them at /etc/lib/mysql/db_name/.
EDIT :
I Searched around internet and I found out that every table should have 3 files related to it.
For example, the table table_users has:
-- table_users.frm
-- table_users.MYD
-- table_users.MYI
and for those hidden table , there are only .frm files and the other two files of a table are missing.
I should change my question to: How to recover a innodb table from a .frm file?
InnoDB does not have those three files
InnoDB data is stored in "ibdata1" for all databases and tables.
the table definition is stored in "tablename.frm"
I would say that your InnoDB file has become corrupted, you may want to have a look at these tools:
https://launchpad.net/percona-innodb-recovery-tool
UPDATED
First of all, about the files:
.frm - table structure;
.myd - table data;
.myi - indexes.
To recover tables, you can try (make backup first):
1) run check table tablename - for all db tables;
2) run repair table tablename - for necessary tables.
UPDATED ONCE AGAIN
Another idea... Try this:
Create a new database to restore and create the tables with same name as .frm files (with the one field - only to create new .frm files);
Stop mysql service and replace the created .frm files with yours;
Start mysql service and check.
I expect correct tables (without data, of course). And sorry, for now I have no PC to check, before suggesting...
actually me too was having the same problem with the missing two files. later i found that when the table's type is innodb then the database folder would have only one associated file.
but you can change the table type to myisam to get all three file for the table.
now as per the backup, you can export the database whenever and wherever you want :)
PHP is GREAT :)
![innodb image][1] INNODB SYSTEM TABLESPACE
INNODB system tablespace is contain in the mysql data directory---
INNODB is system tablespace is divde into two parts
1>.frm
it can describe the table format or you can say it is a table *definition*
2>.ibd
it is contain all system related file and it is also contain data and index and ( InnoDB main table space contain – ibdata1 – and redo logs – ib_logfile*.)
ibdata1 contains your InnoDB database and ib_logfile0 and ib_logfile1 are log files for InnoDB.
If you delete your ibdata1 file, then all your InnoDB tables will be lost.
By default, InnDB uses a shared "tablespace," which is one or more files from a single logical storage area. All InnoDB tables are stored together within the tabespace (from all the databases). By default, InnoDB creates two 5MB log files in the data directory: iblogfile0 and iblogfile1. The information is logged in circular fashion, with old information at the front of the log being overwritten when the log fills up.. Consequently, a larger log allows InnoDB to run longer without having to force changes recorded in the logs to be applied to the tablespace on disk.