My ibdata1 file for MySQL database grew to about 32GB over time. Recently I deleted about 10GB of data from my databases (and restarted mysql for good measure), but the file won't shrink. Is there any way to reduce the size of this file
The file size of InnoDB tablespaces will never reduce automatically, no matter how much data you delete.
What you could do, although it is a lot of effort, is to create one tablespace for every table by setting
innodb_file_per_table
The long part about this is, that you need to export ALL DATA from the mysql server (setting up a new server would be easier) and then reimport the data. Instead of one single ibdata1 file which holds the data for each and every table, you will find a lot of files called tablename.ibd which hold the data only for one single table.
Afterwards:
When you then delete a lot of data from tables, you can let mysql recreate the data-file by issuing
alter table <tablename> engine=myisam;
to switch to MyIsam (and have the InnoDB data file for this table deleted) and then
alter table <tablename> engine=innodb;
to recreate the table.
Unless you set innodb_file_per_table, ibdata1 holds all InnoDB tables plus undo.
This file never shrinks.
To shrink it, you should (at your own risk):
Backup and drop all InnoDB tables in all databases
Delete the file manually
Reinitialize InnoDB storage (by restarting mysqld) and
Restore the tables from backup.
If you set innodb_file_per_table, you'll still have to do this to reclaim the space, but in this case you'll be able to do this on per-table basis, without affecting the other tables.
Note that the undo is still held in ibdata, even with innodb_file_per_table set.
Adding, Removing, or Resizing InnoDB Data and Log Files
Run optimize table your_db.your_table; sql request
or use mysql workbench migration wizard and it will create database copy with reduced size
Related
I need your advice. Need to reclaim disk space on live server with minimum downtime. We are using:
mysql -- 5.5
innodb table per file -- on
Was a huge table(70% insers/30% deletes -- it means that sometimes we delete rows from this table), this table was dropped with "drop table" command, as we expect, mysql didn't release disk space to OS, but now we need to release free space. OPTIMIZE command is possible way -- but now we don't 100% sure if we will have enough free space on disk to do this command, because previous table was huge. But in mysql documentation written:
You can run OPTIMIZE TABLE to compact or recreate a file-per-table tablespace. When you run an OPTIMIZE TABLE, InnoDB creates a new .ibd file with a temporary name, using only the space required to store actual data. When the optimization is complete, InnoDB removes the old .ibd file and replaces it with the new one.
I highlighted confusing moment in documentation, does it means, that "OPTIMIZE dropped_table;" will use only space for data that really needed, for this situation there is no actual data if we drop table early ?
If the table was created when innodb_file_per_table was OFF, then the table was written to ibdata1, and the space is not returned to the OS by DROP TABLE.
If it had been ON, then there would be a .ibd file. Did you see that file before the DROP? Does it still exist? (It should not still exist.)
When DROP removes the .ibd file, there is a slight lag (depending on the OS) while waiting for the data to be actually freed up by the OS. du and/or df reflects this lag.
OPTIMIZE TABLE will copy the table to tmpdir (I think), then drop the old table and play RENAME games. This requires extra space -- enough for a compete copy of the table. Assuming you really have a .ibd file, it will create another .ibd file (with a temp name) and shuffle files. If tmpdir points to a different "filesystem", the shuffle will involve a copy, not just a move.
If I have not explained things adequately, please provide more details -- actual commands you are proposing, size of table in question, whether the .ibd exists, etc.
Can you create a copy of a table by simply making a duplicate of .frm file?
I intend to do something like this:
Suppose my database's name is mydb and it has a table called mytab.
Browse to the folder named mydb in the file system and it has a mytab.frm file.
Then copy the contents of mytab.frm into a file called copy.frm
Then i login to mysql and run the following commands:
use mydb; //Selects the database mydb
show tables; //To see the list of tables. I can see the table named copy.
select * from copy; //This throws the error mentioned in the title.
So what am I missing? What files do you copy to take the backup of a database?
I know that table can be copied by a couple of sql statements But i want to learn something knew so I am experimenting around. Thanks! :)
Just throwing this down as an answer:
You can't do what you have tried for InnoDB tables. InnoDB stores all table data in a single file - ibdata. You can modify this with the innodb_file_per_table setting in my.cnf, but it isn't retroactive, it will only apply to new tables. Even if you DO have file per table setup, you still shouldnt try and just copy the data files, because innodb may not have flushed all changes from the ib_logfile's to the ibdata / .ibd file, so you could well end up with corrupt data.
You can do it for MyISAM tables but you shouldn't (and there are also other files that to be copied as well, as the .FRM is only the table definition. The .MYD file contains the data and the .MYI file contains the indexes). Why? because you are entrusting your data to a database, you should be using database tools to duplicate it. The only time you should be touching the data files directly is during data recovery, and only when the server is not running - you dont want to be copying the files as they're being written to.
To duplicate a table, simply do this:
create table new_table as select * from old_table
To backup an entire database, use mysqldump or one of the other available backup tools.
TL;DR
Copying/moving MySQL tables by altering the underlying files is possible in some conditions but it is highly unrecommended.
Always use MySQL commands to do it.
The .frm file contains only the table definition. The data and the indexes are stored in other files and they depend on the storage engine of the table.
Several excerpts from the official documentation:
MyISAM
15.2 The MyISAM Storage Engine
Each MyISAM table is stored on disk in three files. The files have names that begin with the table name and have an extension to indicate the file type. An .frm file stores the table format. The data file has an .MYD (MYData) extension. The index file has an .MYI (MYIndex) extension.
InnoDB
14.1 Introduction to InnoDB
By default, with the innodb_file_per_table setting enabled, each new InnoDB table and its associated indexes are stored in a separate file. When the innodb_file_per_table option is disabled, InnoDB stores all its tables and indexes in the single system tablespace, which may consist of several files (or raw disk partitions).
14.2.15.1 Role of the .frm File for InnoDB Tables
MySQL stores its data dictionary information for tables in .frm files in database directories. Unlike other MySQL storage engines, InnoDB also encodes information about the table in its own internal data dictionary inside the tablespace. When MySQL drops a table or a database, it deletes one or more .frm files as well as the corresponding entries inside the InnoDB data dictionary. You cannot move InnoDB tables between databases simply by moving the .frm files.
14.12 InnoDB Startup Options and System Variables
innodb_file_per_table
When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table in a separate .ibd file, rather than in the system tablespace.
MySQL Glossary
system tablespace
One or more data files (ibdata files) containing the metadata for InnoDB-related objects (the data dictionary), and the storage areas for the undo log, the change buffer, and the doublewrite buffer. Depending on the setting of the innodb_file_per_table, when tables are created, it might also contain table and index data for some or all InnoDB tables. The data and metadata in the system tablespace apply to all the databases in a MySQL instance.
Prior to MySQL 5.6.7, the default was to keep all InnoDB tables and indexes inside the system tablespace, (...) In MySQL 5.6.7 and higher, the default is file-per-table mode, where each table and its associated indexes are stored in a separate .ibd file.
Let's draw some (partial) conclusion
Before anything else you have to stop the MySQL server (to be sure all the data is safely stored into files).
If the table you want to copy uses the MyISAM engine then you need to copy/rename the .frm, .MYD and .MYI files having the same name as the table.
If the table uses the InnoDB engine and at the moment when it was created the innodb_file_per_table setting was ON then you need to copy/rename the .frm and .ibd files having the same name as the table.
If the table uses the InnoDB engine and it was created while the innodb_file_per_table setting was OFF then you cannot copy or move the table data from outside MySQL.
If the table uses the MEMORY table then it's enough to copy the .frm file and restart the server. The table data and indexes are stored in memory, there is no file for them and the source table will be empty after the server restart, so you get an exact copy of an empty table ;-)
But wait, there is more!
MySQL implements several other storage engines that are probably less used than the ones mentioned above. Each of them has its own rules of storing the data in files.
And more
If the server you want to hack this way is part of a replication cluster the changes you do either are ignored (do not propagate to the other servers in the cluster if you change a slave server) or break the replication (the slave servers are required to query and update a table they don't have, if you change the master server).
The conclusion
Even if, in certain conditions, copying or moving a table by changing the underline files is possible, it is strongly not recommended.
The correct (and many times the only) way to copy a table is to use the commands provided by MySQL.
13.1.14 CREATE TABLE Syntax
Use LIKE to create an empty table based on the definition of another table, including any column attributes and indexes defined in the original table:
CREATE TABLE new_tbl LIKE orig_tbl;
The copy is created using the same version of the table storage format as the original table. The SELECT privilege is required on the original table.
You can then use:
INSERT INTO `new_tbl` SELECT * FROM `orig_tbl`
to copy the data.
Another way
An alternative way to copy a table without writing SQL commands is to export the table definition and data using mysqldump, open the export file in a text editor, change the table name in all the places where it appears, save the file and import it into the database using the mysql command line tool (or other MySQL client).
You have to copy 3 files: copy.frm copy.MYD copy.MYI
Make privileges for files, owner and group
chown mysql.mysql copy.*
chmod 660 copy.*
and refresh tables in mysql :
mysql DATABASE
mysql> flush tables;
and voila!
I have a 20gb MySQL database which contains about 30 million rows of data in one table. I wanted to delete a column within that table to clear up some space.
I dropped the column via phpMyAdmin which took about 15 minutes and checked the disk usage in my server. It went from 20gb to 28gb used after deleting a column that was filled with data.
Did I do something wrong? Am I missing something obvious?
I did nothing else while I was trying to delete the column.
Update: Actually, when I go to the database summary page that lists the tables in phpMyAdmin, it also lists their size. The size of the tables only adds up to 10gb.
When I check the size being used on my server via df -h it says /dev/vda is using 28gb.
The only other files on the server are a few small PHP files. What is taking up this 18gb gap?
Most likely explanation is that you are not using file-per-table mode, and the ibdata1 file that contains all tables and indexes increased during the ALTER TABLE. When you delete a column, MySQL has to create a new copy of the table, move all the data into it (minus the column you deleted) and then drop the original table. Near the end of this process, two copies of the table (minus the column you deleted in one copy) need to be stored simultaneously. Then when that's complete, it drops the original copy and that space can be reused in the future.
Unfortunately, there's no way to shrink the ibdata1. It remains at a size equal to its high-water mark. To recover the disk space, you have to:
Dump all InnoDB tables (or convert them to MyISAM temporarily)
Shut down MySQL Server
Remove ibdata1 and ib_logfile*
Set innodb_file_per_table=1 in your my.cnf
Restart. The missing ibdata1 will be created at the initial default size.
Import your data.
Now the tables will occupy a separate file per table. When you ALTER or OPTIMIZE them, they will recover space. This mode is enabled by default in MySQL 5.6.
I have a table called path. It takes up about 99% of a 13G ibdata1. It was previously an INNODB table, but I converted it to MYISAM.
If I run optimize table on the new path table, will it free up my ibdata1 file? Or does this never reduce in size and I need to delete it and re-import a fresh database?
No. Dropping the InnoDB table will free up space within the InnoDB tablespace (ibdata1 file), but it will not shrink the ibdata1 file.
The exception is that if the table was created while the server innodb_file_per_table variable was set, then the table will be in its own separate InnoDB tablespace (datafile), and when the table is dropped, the space used by the table will be released.
See: 14.3.3. Using Per-Table Tablespaces http://dev.mysql.com/doc/refman/5.5/en/innodb-multiple-tablespaces.html
The answer is no (better explanation above). What I had to do was this:
1) Dump the database
2) Stop mysql
3) Delete the `ibdata1` file and two log files.
4) Restart mysql
5) Import the sql dump.
This will get you back to the default ibdata1 file size. Then you can either 1) change the engine of the table to myisam if there are no integrity constraints on the table (which is what I did) or 2) set innodb_file_per_table=ON, if keeping the innodb table.
Is it possible the create a unique ibdate file for each database made in InnoDB ?
Because if I have multiple database made in InnoDB it will all store it in the ibdata1, ibdata2, etc...
Which make it difficult to just restore 1 database and not all clients who have InnoDB. MyISAM create multiple .frm, MYD and MYI file for each table which make it easier to restore.
How can we make in sort that restore is more easy on InnoDB. Do I have to make a DUMP .sql file or there's other solution ?
Thanks
You can't separate InnoDB data by database, but you can separate it by table, and the tables are stored under the respective subdirectory just like MyISAM tables would be. See http://dev.mysql.com/doc/refman/5.1/en/innodb-multiple-tablespaces.html
But regardless of whether you use a central tablespace file or file-per-table, you should not back up or restore InnoDB databases by moving files around on the filesystem. Even when you're not running queries to write to InnoDB tables, there are background threads that are processing the ib_logfile, undo logs, insert buffers, and other things. Your data is spread in multiple places at any given time, and if you try to move InnoDB files around, you will corrupt them.
Instead, use mysqldump to make logical dumps of InnoDB data safely while MySQL is running.