Innodb; multiple data directories - mysql

I have a requirement to store two Innodb database files on separate partitions. (one database on raid0 for speed, the other on raid1 for stability)
From what I've read, the only way to accomplish this is by using innodb_file_per_table and symlinking .ibd and .frm files wherever-they-may-live to the configured mysql storage directory (/var/lib/mysql on my system), where the ibdata1 file must live (for table metadata).
Is this the only way to accomplish the split?
Is there an innodb analog to myisam's TABLE/INDEX DIRECTORY?

Just to update this post if someone ever come across this, InnoDB now support data directory since version 5.6. No symlink (not recommended), and works for both *nix and Windows.
Requirements:
MySQL 5.6 and above
innodb_file_per_table is enabled
innodb_file_per_table = 1
Scenario 1 (create new tables):
It is as simple as specifying the DATA DIRECTORY options during creation of table.
CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/alternative/directory';
Scenario 2 (moving existing table to separate disk):
This involves few more steps (MySQL server restart required) but still pretty straight forward. And it doesn't requires the table to have DATA DIRECTORY option specified during table creation.
Stop MySQL server
Move the innodb table table_name.ibd file to separate disk/volume by simple file copy/file move
Create a text file with .isl extension e.g. table_name.isl
Edit the .isl file and type in the new path to the table_name.ibd file that you moved.
/alternative/directory/table_name.ibd
Make sure the original .ibd file in old path was removed
Start MySQL server
Now subsequent changes to the moved table will be save into .ibd file in new path.
For reference please refer to MySQL official doc: http://dev.mysql.com/doc/refman/5.6/en/tablespace-placing.html

I just wrote a blog post on this today:
http://www.mysqlperformanceblog.com/2010/12/25/spreading-ibd-files-across-multiple-disks-the-optimization-that-isnt/
You don't want to do the symlinking thing - and InnoDB does not support DATA DIRECTORY/INDEX DIRECTORY.
You actually need stability of the data system wide for InnoDB. Let's say -
You have critical tables A,B,C.
Non-critical tables are D,E,F.
If you relocated D,E,F to /mnt/RAID0, and you lost this volume, InnoDB will actually not allow you to access tables A,B,C any more either.
Unless it can access all your data, InnoDB will refuse to start.
If you have two very different data requirements, you need two installs of MySQL ;)

Scenario 2 MOVE 1 TABLE TO NEW DATA DIRECTORY
Platform - Windows Server,MySQL 5.6
Database - Test
Table - Employee
Source Data directory - D:\Program Files\MySQL\MySQL Server 5.6\data\test\
Destination Data directory -E:\data\New_data\test\
STEPS
Ensure the global variable innodb_file_per_table = 1
Stop MySQL server
Move the innodb file employee.ibd file to separate disk/folder by simple file move to E:\data\New_data\test\
Note - Keep the employee.frm in the original data directory D:\Program Files\MySQL\MySQL Server 5.6\data\test\
in the source data directory - D:\Program Files\MySQL\MySQL Server 5.6\data\test\ - Create a text file with .isl extension e.g. employee.isl
Edit the .isl file and type in the new path to the employee.ibd file that you moved.
In the File -D:\Program Files\MySQL\MySQL Server 5.6\data\test\employee.isl
E:\data\New_data\test\employee.ibd
grant write permission to teh folder E:\data
Start MySQL server
use test;
select * from employee

Related

InnoDB: Cannot open table from the internal data dictionary of InnoDB though the .frm file for the table exists

I am running Windows 10 with XAMPP and several dozen Drupal sites installed on localhost. Everything has been working fine for months.
This morning I performed a Windows restore from a restore point two days ago to get rid of an unwanted Windows update. After I did that, my MySQL stopped working. I tried deleting the file ibdata1 (I now know that that was a bad idea), but when things got worse, I restored the initial ibdata1 that I'd deleted. All the table data (.frm and .ibd files) is still in C:\xampp\mysql\data. Now MySQL will at least start, but all the tables are "gone"... I can load phpMyAdmin in the browser, and when I drop down the databases at left, all the tables show... but when I try to click on one, it tells me "table not found."
In mysql_error.log there are several errors like InnoDB: Cannot open table mysql/slave_master_info from the internal data dictionary of InnoDB though the .frm file for the table exists. There is a URL mentioned in the error message that supposedly tells you how to resolve this issue, but it is uninformative.
I don't think ibdata1 is corrupted... and all the articles regarding recovering data are complicated to a degree that is insanely out of proportion to the problem. No one has a simple spelled-out solution. Am I Googling wrong? This has happened to me at least twice before, and each time the problem spontaneously resolved itself before I was able to start following the 10-hour-long procedures I found online. Not this time, unfortunately. Surely I'm missing something simple. I had hundreds and hundreds of tables in a few dozen databases, and it seems hopeless to try to find the schema for each one and manually plug it in, which is what most recovery articles seem to suggest.
Shouldn't the database just know how to read the .frm and .ibd files, and, you know... work? It feels like there's just one value out of place somewhere, and if I could just find it, everything would just "pop" back into place.
What did the work for us was this link (Restoring Orphan File-Per-Table ibd Files).
In brief, you do:
ALTER TABLE ..... DISCARD TABLESPACE;
For each affected table.
Then, for each affected table:
ALTER TABLE .... IMPORT TABLESPACE; SHOW WARNINGS;
You could do that on the same schema or create a new schema in other database and copy all *.ibd files. In this case:
Create the new schema with table definitions.
Do the discard tablespace for all tables.
Copy the new *.ibd files to the right folder of new schema.
Do de import tablespace for all tables.
IMPORTANT! It's better that the new schema has the same name as the old one.
What worked for me:
https://dba.stackexchange.com/a/42932
My dbs was InnoDB, so there is a problem with id and when upgrading the macos, I got this. I will quote the main part:
You must have:
-ibdata1
-ib_logfile0
-ib_logfile1
-.FRM files from your mysql_database folder
-Fresh installation of MAMP / MAMP Pro that you are willing to destroy (if need be)
SSH into your web server (dev, production, no difference) and browse to your mysql folder (mine was at /var/lib/mysql for a Plesk
installation on Linux)
Compress the mysql folder
Download an archive of mysql folder which should contain all mySQL databases, whether MyISAM or innoDB (you can scp this file, or move
this to a downloadable directory, if need be)
Install MAMP (Mac, Apache, MySQL, PHP)
Browse to /Applications/MAMP/db/mysql/
Backup /Applications/MAMP/db/mysql to a zip archive (just in case)
Copy in all folders and files included in the archive of the mysql folder from the production server (mt Plesk environment in my case)
EXCEPT DO NOT OVERWRITE:
-/Applications/MAMP/db/mysql/mysql/
-/Applications/MAMP/db/mysql/mysql_upgrade_info
-/Applications/MAMP/db/mysql/performance_schema
In my case, I was restoring the backup (done by galera-arbitrator and xtrabackup. Using the first one we took state-snapshot and using the last one we backed up the physical data) for Percona_XtraDB_Cluster. So, after placing the backed up data to the dir /var/lib/mysql of the new mysql server, we did the following select query and got error.
$ mysql -uroot -p$MYSQL_ROOT_PASSWORD -h<NEW_SERVER_HOST_NAME> -e 'show columns from <MY_DB>.<MY_TABLE>'
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1146 (42S02) at line 1: Table 'playground.equipment' doesn't exist
And the server log was:
[Warning] InnoDB: Cannot open table <MY_DB>/<MY_TABLE> from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
We just restarted the container the new data. That's it
Just rename slave_master_info.frm to slave_master_info.BAK and that's it. Restart the mysql service.

How do I make my databases in MySQL go on another drive on my computer?

Say my website takes in lots of data from its users, and it goes to MySQL on my computer, but my computer runs out of space. If I connect a hard drive to my server computer, can I make it so I can put a new database on that hard drive, and all the data gets stored there? It would obviously be attached at all times.
This probably belongs on https://dba.stackexchange.com/ instead, and indeed there's quite a bit of in-depth discussion there about some of the techniques you could use.
One thing I've found helpful when working with large but temporary datasets is to enable innodb_file_per_table which — in my case — helps reclaim disk space when removing these temporary databases.
Moving the entire datadir
You can move the entire directory that MySQL uses to store files; this is called the datadir. Stop the MySQL daemon, move the folder, edit my.cnf to refer datadir = to the new folder location, and start the daemon.
File-per-table tablespace outside of the datadir
https://dev.mysql.com/doc/refman/5.6/en/tablespace-placing.html
You can use the file-per-table tablespace configuration (innodb_file_per_table in the [mysqld] portion of the configuration file) in conjunction with CREATE TABLE to actually place a tablespace outside the datadir. When you have innodb_file_per_table enabled, you can use statements like CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/alternative/directory'; to put the datadir for that tablespace in a different directory.
Partitioning
Using partitioning, you can break up the databases, tables, and columns in to different storage partitions. This is probably not what you're looking for in this case because they still reside in the datadir. https://dev.mysql.com/doc/refman/5.7/en/partitioning.html
You can export the database via PhpMyAdmin and the pass the saved database file onto another computer.
Exporting the database:
https://serverpilot.io/community/articles/how-to-export-a-database-using-phpmyadmin.html
Importing the database:
https://serverpilot.io/community/articles/how-to-import-a-database-using-phpmyadmin.html

Restoring MySQL Databases when reinstalling Wamp

I am attempting to restore my old databases to a new Wamp installation. However, I did not take a backup of Mysql files before I did this. Thankfully the uninstallation has not deleted all the files, but in my data folder I have :
1) All the databases (As folders)
2) auto.cnf
There are no ib* files in the old folder. Is it still possible to restore my databases somehow?
I have tried to replace the entire mysql5.6.17 folder (there has been no change of version between the installations), but Wamp remains orange and never turns green.
If I replace only the data folder, the MySQL service starts, I can see the databases, but not the tables.
If I change the path in my.ini to point to the old wamp's data folder, I cannot see the old databases at all.
EDIT : Was experimenting a bit, copying the database folders directly into the new Wamp's data folder allows me to get a list of all the tables properly, but there is no actual data in the tables, and trying to query it using MySQLWorkbench gives me the error :
Error Code: 1146. Table 'testdb.users' doesn't exist
EDIT 2 : Found this in the error log file in my data folder :
2014-08-03 06:24:46 25164 [Warning] InnoDB: Cannot open table testdb/users from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
As I thought, its a problem with InnoDB's own mapping. Is there a way to resolve this manually?
If you created an INNODB database, and made no other config changes to store the actual data in seperate table spaces then all the actual data for that INNODB databases will have been stored in the \wamp\bin\mysql\mysql5.6.17\ibdata1 file.
If you do not have that file from the old install then your data is lost.
There will also be info in a \wamp\bin\mysql\mysql5.6.17\data\DATA_BASE_NAME\ folder, one .frm file for each table that was in the database, but this is basically just schema information and not actual data.
For future reference, InnoDB Backup and Recovery
GOLDEN RULE:
Always backup database (from phpmyadmin) before major changes.. better to spend 1 minute for that, than going into trouble..
How to Copy data folder
If you want to copy the database, then INSTALLATION/mysql versions should be same.. then rename newly installed data folder to i.e. data_my_old,so now there you can place the backuped data folder (it should contain ibdata1 file).
Depends on innodb_file_per_table setting.
If it was OFF then all your tables were in ibdata1. Now you have quite new ibdata1 (since you see errors "Cannot open table testdb/users from the internal data dictionary", that means InnoDB runs fresh new dictionary). That means that now the data is somewhere on the disk. The recovery is quite difficult and requires a lot of tedious work. You need to take an image of the disk, scan it with stream_parser to find InnoDB pages, then recover the InnoDB dictionary, then find index_id of every table and if the stream_parser has found this index run c_parser to extract records from the index.
If innodb_file_per_table was ON and you do see .ibd files in the databases directories then recovery is much easier. First, you may try to reconnect individual ibd files with ibdconnect or extract records from the ibd files with stream_parser/c_parser.
The mentioned tools are part of TwinDB data recovery toolkit
i have just done a google search and found this....
Go to C:\wamp\bin\mysql\mysql5.5.24\data
Find a file called "mysql-bin.index"
Rename that file to "mysql-bin.index-corrupt"
Restart the server and it will work fine, all your database will be where you left them.

Not all tables showing up when recovering MYSQL

I am recovering a missing mysql db with tables and have the frm files and ibdata1.
I have used innodb_file_per_table to make each DB have its own data file and tests show that new DB now use this system when created.
I noticed that when I to view the newly imported frms and data or write any SQL against them, they aren't being found. Several 'views' are available, but no tables.
If I run check table sometablename, I get that the table doesn't exist.
If I try to create a table with the same name, I get that the table already exists.
I have stopped and started the MYSQL service several times, hoping it would pick up the new frms, but nothing.
Any idea?
You can't restore a database by simply loading in the contents of the database directory. InnoDB stores other information in some master files located in the main MySQL data directory.
The only reliable method of recovering all data is to either use a mysqldump and restore procedure, or to FLUSH WITH WRITE LOCK before copying the entire contents of the MySQL data directory. This method is not recommended for production systems as it locks the entire database for the duration of this operation.

MySQL InnoDB database restore

I have to restore a database that has been inadvertently DROPped in MySQL 5.0. From checking the backup files, I only seem to have .FRM files to hold the database data.
Can anyone advise whether this is all I need to perform a database restore/import from the backup, or are there other files I should have to hand to complete this?
.frm files are not the data files, they just store the "data dictionary information" (see MySQL manual). InnoDB stores its data in ib_logfile* files. That's what you need in order to do a backup/restore. For more details see here.
Restoring innodb:
(assuming your data folder is C:\ProgramData\MySQL\MySQL Server 5.5\data)
Copy the folders of the databases (named after the database name) you want to restore to C:\ProgramData\MySQL\MySQL Server 5.5\data
Copy the 3 ibdata files to the data folder ex. (C:\ProgramData\MySQL\MySQL Server 5.5\data)
_ib_logfile0
_ib_logfile1
_ibdata1
Get the size of the _ib_logfile0 in MB (it should be the same as _ib_logfile1) by File Right click -> Properties
Edit the mysql config file (mysql\bin\my.ini) for the innodb_log_file_size=343M to be exactly the ibdata files size
Run
mysqld --defaults-file=mysql\bin\my.ini --standalone --console
--innodb_force_recovery=6
Now your data should be back in your database. Export them using phpmysql or any other tool
The detailed solution you can found here:
http://www.unilogica.com/mysql-innodb-recovery/ (Article in Portuguese)
Besides the flag of innodb_force_recovery, I found another solution: innodb_file_per_table, that splits InnoDB tables in each file like MyISAM tables.
In a crash recovery you can lost less data than in single file ibdata1.