I was running an optimize on a 70GB+ data table (after adding a bit column), and the database just crashed.
When browsing the data folder, I have a [table].ibd file and a #sql-*.ibd orphan file.
The only way I could restart mysql daemon was to start it with innodb_recovery_mode = 6.
But when reading the table, the columns have values that do not make any since.
One of the columns that is of type BIGINT just repeats the same number for rows on end...
I tried do DISCARD TABLESPACE and the IMPORT it again - it crashes the service while doing it.
InnoDB warns about corrupted table and also corrupted indexes.
I'm looking for help to try and recover the data, since it is very important.
Is it possible to read the .ibd file and recover the data?
I also used "undrop-for-innodb" to try to recover the data, but the values in the rows make no sense.
Thank you in advance.
Related
MySQL server suddenly stopped working and the service would go on and off and I could not take backup as the service would not run so best I could do is Copy the whole DATA folder .
Also I had a earlier backup so using that along with the ibd files i restored all the table except one for one last table its saying :
ERROR 1808 (HY000): Schema mismatch (Clustered index
validation failed. Because the .cfg file is missing, table
definition of the IBD file could be different. Or the data file
itself is already corrupted.)
And I have no clue what to do.
I think the only thing you can do is rely on the version of the table from your backup. You cannot restore the .ibd file you copied.
The error message is saying one of two things happened:
The table structure changed since the backup was made, so the .ibd file no longer matches the metadata stored in the data dictionary. When the content of the tablespace file doesn't match the metadata, InnoDB is in the same boat as you are: "I have no clue what to do."
Even if the table structure did not change, the .ibd file was physically corrupted enough that it cannot be read by InnoDB.
Either way, that tablespace file cannot be read by InnoDB.
Obviously this creates a problem if you restored all the other tables successfully. Now you have more current data in most of your tables, except for the last one, which is outdated. If there are any rows in these tables that reference each other, they may have orphaned records (for example, a record showing a certain user bought a product, but the user doesn't exist in the users table).
That's unfortunate, and difficult to correct.
In general, copying .ibd files is not a reliable way of backing up an InnoDB database. You need to use proper backup tools like mysqldump or Percona XtraBackup.
Another solution to restore data after your last backup is to use point in time recovery using binary log files. But for that to work, you would need all the binary log files since your most recent backup, and the backup would need information about the binary log position when the backup was taken. See https://dev.mysql.com/doc/refman/en/point-in-time-recovery.html
I am a newbie to Mysql/Mac OS. Yesterday I turned my laptop (Yosemite 10.10.5) on and found that mysql workbench 6.3 was complaining when retrieving data from one of my tables:
Error Code: 1932. Table 'mydb.mytable' doesn't exist in engine
I cant think why this would suddenly start happening. The ibd file is 2.4GB but I guess that makes no difference.
The table is stored in INNODB format.
All I need to do is get the data, even if its in a text file or something, I dont care about relinking the data back into MySQL. At this stage all I care about is the data.
I've tried various things:
innodb_force_recovery (values from 0-6) (same error occurred)
PERCONA innodb recovery tool (i couldnt get it to compile)
deleting and recreating tablespaces (same error occurred)
http://www.chriscalender.com/tag/innodb-error-tablespace-id-in-file/ - but this does not give me the error he says it should, hence he thinks I should be able to dump my table (which I cant) as the same error occurs.
I'm at my wits end. Is there any way to get my data back? Any advice much appreciated.
How can I restore a partitioned mysql InnoDB table from just the .ibd files of the form TableName#P#pname.ibd ? Chris Calendar's article here
http://www.chriscalender.com/?p=28
works for non-partitioned tables with a single .ibd file, but the "discard" and "import" steps result in a "storage engine doesn't have this option" error for partitioned tables.
Wes Smith in one of the comments at the link above suggests a manual procedure to import one partition at a time, but that did not work for me. If I try to follow his approach by creating a non-partitioned table, moving the first .ibd file renamed as TableName.ibd, and doing an import, the import succeeds, but there are zero rows in the table. The subsequent suggested step of "add back in the partition", which I tried as "alter table TableName partition by range ... (partition pname1 values less than (...) ENGINE=InnoDB)"shockingly replaces the TableName.ibd file (corresponding to the first partition) with a fresh TableName#P#pname1.ibd of a trivial size. I lost one partition's worth of data trying this. I have about 150 partitions to recover.
Any advice on how to recover the data from the .ibd files? Thanks.
So, it turns out that the approach suggested in the comment at the link in the original post does work after all. The deletion of the partition I mentioned in the "alter table..." step above was legitimate as it had been intentionally deleted in the original table before the loss of the ibdata and log files (but mysql leaves the .ibd files undeleted anyway). The recovery process is painfully slow, but I have managed to script it up and hope to let it run to completion over the next several days.
Here are some tips if you find yourself in a similar situation with many partitions to recover. Suppose you had a table TableName with 100 partitions. Typically, the 100 partitions would have consecutive innodb IDs if they were created via a "create table" statement, but in general this may not be the case as partitions may have been added after creation. So, here are the steps:
1) Find out the innodb ID corresponding to each partition using Calendar's method in the blog above as follows. Note that this step does not need a restart of the mysql server. Just create a table like TableName without any partitions and discard its tablespace. Then move the first partition's .ibd file (named something like TableName#P#pname1.ibd) to the database directory as TableName.ibd and try to import it. Look into the mysql error.log (typically /var/log/mysql/error.log) to see what that partition's innodb ID is. Repeat this step for each partition (or as needed) until you have the 2-tuples (innodb_ID_i, partition_boundary_i) for all partitions in a file.
2) Start with an empty innodb state (stop the server, delete ibdata and ib_logfile*, restart server). For each innodb_ID entry in the file in step 1 above, create a table TableName_i like TableName. E.g., if the 100 partitions correspond to the IDs 321, 322,..., 370, 415, 416,...464 (two blocks of 50 contiguous IDs each), then write a script to create 320 dummy tables, 50 tables like TableName, 45 dummy tables, and 50 tables like TableName.
3)
For each TableName_i table created above,
--do
(i) rename table TableName_i to TableName
(ii) alter table TableName discard tablespace // important to do this step before the next one
(iii) mv TableName#P#pname_i.ibd TableName.ibd // with the appropriate directory prefixes
(iv) alter table TableName import tablespace
(v) alter table partition by range (partition_field) (partition pname_i values less than (partition_boundary_i)) // This is the most and only time consuming step
(vi) rename table TableName to TableName_i // or some other name or just dump it to a file
--repeat
Note that all of the above steps are scriptable and do not require restarting the server at any point except at the beginning of step 2 to start with an empty innodb state. Be careful to introduce checks for the success of each sub-step in step 3 before moving to the next, otherwise successive steps may fail and/or .ibd files may get overwritten. If feasible, use a copy in step 3(iii) instead of mv.
A final note: There might be a slightly easier alternative using percona's recovery toolkit using hex editing, but this did not work for my case of partitioned tables. I ran into the same, seemingly unresolved issue, with partitioned tables as noted in one of the comments at http://www.mysqlperformanceblog.com/2011/05/13/connecting-orphaned-ibd-files/ . Your mileage may vary though. If there were a way to avoid recreating partitions (like in step 3(v) above), that would be real nice and quick, but I am not sure if there is one.
I have a serious problem in my MySQL tables , once there were InnoDB tables which were IN USE and now are somehow hidden
look at this [pic] *Link removed - the number of tables in heading is 79 and actual counted number is 74.
these tables are those that were IN USE
I don't have any recent backup of my database , so this would game of life and death for me
I checked my VPS, I found them at /etc/lib/mysql/db_name/.
EDIT :
I Searched around internet and I found out that every table should have 3 files related to it.
For example, the table table_users has:
-- table_users.frm
-- table_users.MYD
-- table_users.MYI
and for those hidden table , there are only .frm files and the other two files of a table are missing.
I should change my question to: How to recover a innodb table from a .frm file?
InnoDB does not have those three files
InnoDB data is stored in "ibdata1" for all databases and tables.
the table definition is stored in "tablename.frm"
I would say that your InnoDB file has become corrupted, you may want to have a look at these tools:
https://launchpad.net/percona-innodb-recovery-tool
UPDATED
First of all, about the files:
.frm - table structure;
.myd - table data;
.myi - indexes.
To recover tables, you can try (make backup first):
1) run check table tablename - for all db tables;
2) run repair table tablename - for necessary tables.
UPDATED ONCE AGAIN
Another idea... Try this:
Create a new database to restore and create the tables with same name as .frm files (with the one field - only to create new .frm files);
Stop mysql service and replace the created .frm files with yours;
Start mysql service and check.
I expect correct tables (without data, of course). And sorry, for now I have no PC to check, before suggesting...
actually me too was having the same problem with the missing two files. later i found that when the table's type is innodb then the database folder would have only one associated file.
but you can change the table type to myisam to get all three file for the table.
now as per the backup, you can export the database whenever and wherever you want :)
PHP is GREAT :)
![innodb image][1] INNODB SYSTEM TABLESPACE
INNODB system tablespace is contain in the mysql data directory---
INNODB is system tablespace is divde into two parts
1>.frm
it can describe the table format or you can say it is a table *definition*
2>.ibd
it is contain all system related file and it is also contain data and index and ( InnoDB main table space contain – ibdata1 – and redo logs – ib_logfile*.)
ibdata1 contains your InnoDB database and ib_logfile0 and ib_logfile1 are log files for InnoDB.
If you delete your ibdata1 file, then all your InnoDB tables will be lost.
By default, InnDB uses a shared "tablespace," which is one or more files from a single logical storage area. All InnoDB tables are stored together within the tabespace (from all the databases). By default, InnoDB creates two 5MB log files in the data directory: iblogfile0 and iblogfile1. The information is logged in circular fashion, with old information at the front of the log being overwritten when the log fills up.. Consequently, a larger log allows InnoDB to run longer without having to force changes recorded in the logs to be applied to the tablespace on disk.
My ibdata1 file for MySQL database grew to about 32GB over time. Recently I deleted about 10GB of data from my databases (and restarted mysql for good measure), but the file won't shrink. Is there any way to reduce the size of this file
The file size of InnoDB tablespaces will never reduce automatically, no matter how much data you delete.
What you could do, although it is a lot of effort, is to create one tablespace for every table by setting
innodb_file_per_table
The long part about this is, that you need to export ALL DATA from the mysql server (setting up a new server would be easier) and then reimport the data. Instead of one single ibdata1 file which holds the data for each and every table, you will find a lot of files called tablename.ibd which hold the data only for one single table.
Afterwards:
When you then delete a lot of data from tables, you can let mysql recreate the data-file by issuing
alter table <tablename> engine=myisam;
to switch to MyIsam (and have the InnoDB data file for this table deleted) and then
alter table <tablename> engine=innodb;
to recreate the table.
Unless you set innodb_file_per_table, ibdata1 holds all InnoDB tables plus undo.
This file never shrinks.
To shrink it, you should (at your own risk):
Backup and drop all InnoDB tables in all databases
Delete the file manually
Reinitialize InnoDB storage (by restarting mysqld) and
Restore the tables from backup.
If you set innodb_file_per_table, you'll still have to do this to reclaim the space, but in this case you'll be able to do this on per-table basis, without affecting the other tables.
Note that the undo is still held in ibdata, even with innodb_file_per_table set.
Adding, Removing, or Resizing InnoDB Data and Log Files
Run optimize table your_db.your_table; sql request
or use mysql workbench migration wizard and it will create database copy with reduced size