.ibd files taking too much space on ubuntu server - mysql

Question

I'm having a problem with .ibd MySQL files.
Scenario:
I'm having a ubuntu server of 200GB and deployed an application of Django and using MySQL server.
The nature of my application is to store huge data and do some x type of processing on it. I have one table which has 5 to 6 million data recrods. This Table has acquired almost 60GB of space (The space occupied by tablename.ibd file).
I tried running Optimize table tablename but the .ibd file doesn't get shrunk.
The InnoDb is true.
PROBLEM
Firstly the storage is running out as the file getting too much large.
Secondly when I try to migrate the migration for adding a column on this table while running the server gets out of space because on running migration the .ibd file starts getting bigger and the server eventually runs out of space.
I will be very thankful If someone helps me out of this.
Note:(I could not purge data from the table as data is very important for me)
(UPDATED)
SHOW CREATE TABLE tablename
| Table | Create Table |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| table_name | CREATE TABLE `table_name` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) DEFAULT NULL,
`price` double DEFAULT NULL,
`item_identifier` varchar(20) NOT NULL,
`upc` varchar(20) DEFAULT NULL,
`mpn` varchar(100) DEFAULT NULL,
`weight` double DEFAULT NULL,
`weight_unit` varchar(10) DEFAULT NULL,
`main_category` varchar(50) DEFAULT NULL,
`sub_category` varchar(50) DEFAULT NULL,
`category_tree` varchar(500) DEFAULT NULL,
`description` varchar(3800) DEFAULT NULL,
`color` varchar(50) DEFAULT NULL,
`brand` varchar(150) DEFAULT NULL,
`main_image` varchar(2048) DEFAULT NULL,
`secondary_images` varchar(255) DEFAULT NULL,
`shipping` double,
`stock` int(11) NOT NULL,
`sale_rank` varchar(100) DEFAULT NULL,
`itemHeight` double DEFAULT NULL,
`itemLength` double DEFAULT NULL,
`itemWeight` double DEFAULT NULL,
`itemWidth` double DEFAULT NULL,
`manufacturer` varchar(100) DEFAULT NULL,
`product_model` varchar(150) DEFAULT NULL,
`variations` longtext,
`pack_count` int(11),
`size` varchar(100) DEFAULT NULL,
`flavor` varchar(100) DEFAULT NULL,
`successfully_stored` tinyint(1) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `item_identifier` (`item_identifier`),
KEY `table_name_upc_3ca3d702` (`upc`)
) ENGINE=InnoDB AUTO_INCREMENT=7279139 DEFAULT CHARSET=latin1 |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
SHOW TABLE STATUS LIKE 'tablename'\G
*************************** 1. row ***************************
Name: table_name
Engine: InnoDB
Version: 10
Row_format: Dynamic
Rows: 7439966
Avg_row_length: 8807
Data_length: 65530740736
Max_data_length: 0
Index_length: 323633152
Data_free: 5242880
Auto_increment: 7279139
Create_time: 2021-06-11 21:26:17
Update_time: 2021-06-12 18:08:06
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.01 sec)

Answer 1

InnoDB disk space is 2-3 times as much as you might think. This is because of several different "overhead" things. They provide performance and features; live with it.
60GB / 5M = 12KB per row. Sounds like you have one more big TEXT or BLOB columns? Please provide SHOW CREATE TABLE so we can further discuss the layout of the table.
(OPTIMIZE TABLE is rarely of any use; don't bother using it.)
Sizes
Bill covered most of the size-related things (DOUBLE->FLOAT, etc); alas they will shrink the disk footprint by only a few percent in your case.
It seems that variations must be the bulkiest column. What do you get from SELECT AVG(LENGTH(variations)) FROM table_name; ? I suspect it is a few thousand. Most "text" can easily be compressed 3:1 by standard compression libraries. If the average is 3000, then the potential savings is about 2KB which is something like 20-30% of the table. (It may save more due to the "off-record" storage mechanism, but the computation is complex.)
Compressing a single column requires the cooperation of the client. That is, code in Django needs to compress and uncompress the column between client and server.
Using ROW_FORMAT=COMPRESSED gives about 2:1 compression for the whole table and is transparent to the client. So, overall, this is probably better.
As Bill points, out, all of this is a temporary fix -- you will run out of disk space as the table grows. That is, Optimize, smaller datatypes, and compressions are only temporary fixes. You really need more disk space.

Answer 2

Get a server with larger storage volumes.
Alternative: Get a second server running MySQL Server, and move some of the data in your current instance to that new instance.
Re your update with the table definition and status:
The table status shows that the data length, that is, the rows, use about 61 GiB, and the secondary indexes use about 0.3 GiB. So it's unlikely that you can save space by dropping indexes.
The average row size is 8807 bytes (this is an estimate, it's just the data_length divided by the number of rows). You might be able to reduce the average row size a little bit by changing some data types.
For example, each double takes 8 bytes. Could you get enough precision using float or numeric(9,2) instead? These take 4 bytes each. Similarly, there are some int columns that might be able to be smallint and still store the range of values you need.
You should read about the storage requirements of each data type and make decisions about how best to define your columns. See https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html
The variable-length data types like varchar and longtext already store only the length of the content in the column on each row, not the max length allowed. So for example changing varchar(200) to varchar(100) doesn't make any difference if the strings in them are already shorter than 100 characters.
There are some cases of varchar that might be replaced by an integer reference to a lookup table. An integer may take less space than repeating the same string on every row.
You could use the InnoDB COMPRESSED row format. This has variable results depending on your data, but it might shrink strings by about half.
Changing data types and the row format do require you to run ALTER TABLE, so there needs to be enough storage space for the copy of the table temporarily, similar to running OPTIMIZE TABLE. If you don't have enough space to do that, then you can't alter the table.
Even with these techniques, your table will still be quite large, and databases tend to grow over time as we store more rows of data in them. Even if you shrink it a bit today, you will still need a plan for getting a larger storage volume eventually.