I have several databases in MySQL with InnoDB engine. All together they have around 30 GB size on filesystem. A couple of days ago, I removed a lot of data from those databases (~10-15 GB) but the used space on filesystem is the same and also reading data_length and index_length from information_schema.TABLES give almost the old size.
I dumped a 3,3 GB database and imported it on my workstation where it takes only 1,1 GB (yes, it is a 1:1 copy). So, how can I calculate the size a InnoDB database needs if I would reimport it in a new system?
Optimize your tables after deleting large amouts of data.
http://dev.mysql.com/doc/refman/5.1/en/optimize-table.html
InnoDB doesn't free disk space; it's a PITA. Basically, you can free it by dropping the database and restoring it from a backup (as you've noticed by chance) - see here for examples.
So, you can't calculate how big a database will be after you restore a backup. But it will never be bigger than the un-backed up one (because the unbacked up version still has space from any deleted data and the restored backup will not have that space).
This can be worked around to some extent using the file per table option; more details in the first link from this post.
Related
Using: MySQL 5.6 on Debian 9, total DB size is around 450Gb
Updated to 5.7, ran mysql_upgrade, noticed that around 150 GB has been taken up. 2 tables are really large and they stayed in 'copying to tmp table' for a couple of hours
Noticed innodb_file_per_table was on and created large ibd files that weren't there previously.
Restored from a snapshot, disabled the file_per_table, ran mysql_upgrade again. 100GB gone, which is almost 1/4 of my total DB.
In the first case, it pulled the data from ibdata and put it into a separate file but ibdata never shrinks so taken space almost doubled.
What happens in the second case? Does the temp table get created within the ibdata file that never shrinks so even when table is not used anymore - space is still gone?
Another thing I noticed is that space consumption doesn't start until query has been in copying to tmp table status for like an hour or so.
1) Is there any way to avoid/minimize space increase?
Would running update with file_per_table on, then disabling it and running alter table engine innodb free up the space?
2) Any way to predict how much space will be occupied? At least per table
3) How does max_tmp_table_size play into this?
It sounds like you painted yourself into a corner by not running innodb_file_per_table from the start, so now you have a huge, unshrinkable ibdata1 file.
1) There isn't.
1.1) It might reduce the overall space usage by rebuilding the tables outside the ibdata1 file, then rebuild them again to inside ibdata1, reusing some of the unused space inside ibdata1
2) Yes:
SELECT TABLE_SCHEMA, TABLE_NAME, DATA_LENGTH, INDEX_LENGTH FROM information_schema.TABLES;
3) It doesn't. The tables you are seeing are probably tables being rebuilt for some reason (not sure why, I have to admit I haven't seen that happen from mysql_upgrade before). max_tmp_table_size is only for implicit (when a query plan says using temporary) and explicity (CREATE TEMPORARY TABLE ...) temporary tables, not for table rebuilds.
The only(?) way to switch to file_per_table without the disk bloat is
Dump the data.
Get a fresh install (or otherwise get rid of ibdata1).
Reload (with file_per_table on).
I have largish (InnoDB) tables in a database; apparently the users are capable of making SELECTs with JOINs that result in temporary, large (and thus on-disk) tables. Sometimes, those are so large that they exhaust disk space, leading to all sorts of weird issues.
Is there a way to limit temp table maximum size for an on-disk table, so that the table doesn't overgrow the disk? tmp_table_size only applies to in-memory tables, despite the name. I haven't found anything relevant in the documentation.
There's no option for this in MariaDB and MySQL.
I ran into the same issue as you some months ago, I searched a lot and I finally partially solved it by creating a special storage area on the NAS for themporary datasets.
Create a folder on your NAS or a partition on an internal HDD, it will be by definition limited in size, then mount it, and in the mysql ini, assign the temporary storage to this drive: (choose either windows/linux)
tmpdir="mnt/DBtmp/"
tmpdir="T:\"
mysql service should be restarted after this change.
With this approach, once the drive is full, you still have "weird issues" with on-disk queries, but the other issues are gone.
There was a discussion about an option disk-tmp-table-size, but it looks like the commit did not make it through review or got lost for some other reason (at least the option does not exist in the current code base anymore).
I guess your next best try (besides increasing storage) is to tune MySQL to not make on-disk temp tables. There are some tips for this on DBA. Another attempt could be to create a ramdisk for the storage of the "on-disk" temp tables, if you have enough RAM and only lack disk storage.
While it does not answer the question for MySQL, MariaDB has tmp_disk_table_size and potentially also useful max_join_size settings. However, tmp_disk_table_size is only for MyISAM or Aria tables, not for InnoDB. Also, max_join_size works only on the estimated row count of the join, not the actual row count. On the bright side, the error is issued almost immediately.
I got the free space (fragmentation issues) in my information_Schema database.
Alert shows that there are 1500% free space in some tables like COLUMNS , ROUTINES.
I am worried how this is possible because i don't have any routines in my database and how i can optimize the information_schema because its memory based database and created on the starting of mysql service.
Also when i query "SHOW CREATE TABLE" on any of the information_schema table it gives me innodb as engine of these table, but i think it should be memory.
Any idea to optimize these tables without restart?
Thanks
When you have innodb_file_per_table = OFF, InnoDB tables are created in the system 'tablespace', ibdata1. It could be that you have created and manipulated a lot of tables there.
Data_free is a confusing term in SHOW CREATE TABLE and certain tables in information_schema...
For MyISAM tables, it is an accurate amount of the space that could be recovered from the .MYD file (but not the .MYI file).
For InnoDB it means one of 2 things...
If the table you are looking at was created with innodb_file_per_table = ON, then Data_free is some of the unused space. Often, not all of it can be recovered by any means.
If the table you are looking at was created with innodb_file_per_table = OFF, then Data_free is the free space in ibdata1. That free space will be used for new inserts and new tables, thereby decreasing Data_free. However, the size of ibdata1 cannot be shrunk, at least not without a lot of effort (dump everything, remove, reload).
I have a database and that contain many tables , but all of them store text and full tables size is less than 10 MB , but the database file on server have larger than 1 GB
The tables uses MyIASM , where is the problem ?
thanks
You can use OPTIMIZE TABLE for reducing disk usage on MyISAM tables (usually after deleting lot of data from them or doing major changes in structure).
For InnoDB, that is quite more challenging, see Howto: Clean a mysql InnoDB storage engine?
I'm using InnoDB database with a single file configuration (in /var), so no innodb_file_per_table.
In the MySql workbench, when I query for the databases used space, with this query
SELECT table_schema "Database", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB"
FROM information_schema.TABLES GROUP BY table_schema;
It says that I have 47 GB of data.
However the size of ibdata1 is 99 GB...
I know that ibdata1 contains a bunch of other things other that table data, like Table Indexes, MVCC (Multiversioning Concurrency Control) Data and Table Metadata
So my question is: Is it normal that supposedly 52 GB of ibdata1 is medatada and a bunch of other things? Usually, how much data beside table data should the ibdata1 file contains?
No, it is not normal that you would have that much metadata. It is normal though that the ibdata file can grow to a ridiculous size if you aren't using innodb_file_per_table.
Your ibdata file will grow when your database grows, but it will never actually shrink.
So, for example, if you had 130 GB of data at one point and deleted a bunch of it, the ibdata file would still be 130 GB after the data was purged. It will just have a bunch of "free space" that it will then use for subsequent inserts.
As for shrinking the file, there's not much you can really do aside from wiping out your database and restoring it. This answer has some good instructions on how to do that.
Howto: Clean a mysql InnoDB storage engine?
You also might want to consider using innodb_file_per_table as deleting data from a table and later optimizing that table will actually shrink the size of the individual table files
There are a few reasons for having a bunch of "extra" space in ibdata1, but the most likely cases are:
You have deleted large amounts of data in the past. When you delete rows or drop tables, although free space will be made available in the file, the file itself will never shrink.
You may have an excessive amount of undo log space (or have at some point in the past). Undo logs are kept during DELETE and UPDATE operations, and for very long-running operations touching many rows can grow quite large. Again, if the file is expanded to hold this data it will never shrink.
As previously mentioned using innodb_file_per_table can help with this if you expect to regularly drop tables and want to get the disk space back. My blog post The basics of InnoDB space file layout may help you understand what is included in the ibdata1 file.