I have an innodb table with big table size (around 7GB),
I already delete about 80% of its rows,
The problem is the table size remains the same,
The innodb_file_per_table is ON,
How do I shrink the table size so it reflect the actual condition and save more disk space ?
It depends...
If you built the table with innodb_file_per_table=ON, then you can do OPTIMIZE TABLE now. (Be sure it is still ON.)
If =OFF, then the table is in the tablespace file ibdata1 and the only way to return the space to the OS is by dumping all tables, removing ibdata1, then reloading the dump. This is tedious and risky. Meanwhile, the free space created by the DELETEs will be used by future INSERTs, etc.
More discussion of big deletes.
Related
I have very large mysql DB(more than 100 G) and I want to migrate some changes in table. It need to posing that I have no space to make backup for this size, as results, it rolls back all changes. In clear way, when I want to alter table in mysql, it backup for it self in local disk and because of no space on my disk, it 's rolling back. Is there any one to help for this issue?
It depends on the type of ALTER change. Since MySQL 5.6, some alterations can be done "inplace" which does not use extra storage. Read https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html for details.
But many types of alterations (basically any change that affects the storage size of a row), still require a "table rebuild" which means it makes a copy of the table in the process of doing the alter. This requires extra storage, usually about the same as the size of the original table. If your server does not have enough storage space to do this, then you cannot alter the table, FULL STOP. You should have addressed this long before the table grew to this size. It is your responsibility to monitor the size of the database and make sure you have enough storage space.
You said in the comments above that it is not possible to add more storage space.
Sometimes InnoDB tablespaces can become "fragmented." That means they could store the same rows using slightly less storage space. You can accomplish defragmentation with OPTIMIZE TABLE <name>; or ALTER TABLE <name> FORCE;
A possible strategy is to optimize each of your tables, starting with the smallest table and doing one by one in order of size. You hope that you have enough space for the copy table needed to do this, and any space freed up by optimizing the small tables will help you to optimize the medium-size tables, and so on. This is a long shot, because it's also possible you don't have enough "wasted" space to recover to make a difference. You may still not be able to alter your largest tables once you're all done with this process. Unfortunately, MySQL has no way of reporting fragmentation of the tables, so you can only try it and see how much space, if any, you recover.
After that, the only other option is to delete data from the table until you can do the alteration. The alter still requires extra storage, but only as much as needed to copy the remaining rows of data. That is, if you delete 50% of the rows, then the tablespace file that stores the copy only needs roughly 50% of the storage space.
You can also try dropping some of the indexes of the table as you do your alteration. Indexes take space in the same tablespace, and it's not uncommon if you have multiple indexes for them to take more space than the data itself. If you include some DROP INDEX operations in your ALTER TABLE statement, the resulting copy will take less space. On the other hand, those indexes might be necessary for proper query optimization, and without the indexes, your application won't be usable.
Could you tell me, will MySQL data in tables be physically fragmented if I insert new data and delete old data (insert in table top, and delete bottom rows)?
Will the size of table be growing in any way, before I do OPTIMIZE?
Are you DELETEing and INSERTing at about the same rate? Is the table InnoDB?
If Yes to both, then "don't worry". The DELETEs will free up blocks and the INSERTs will re-use those blocks (or new blocks if the INSERTs are getting ahead).
InnoDB tables are composed of 16KB blocks that are chained together in a BTree structure. What you described will be freeing blocks from one side of the tree and creating new blocks on the other side. However, each block can reside anywhere, so the concept of "side of the tree" is more 'virtual' than 'physical'.
After you have done a lot of churning, try the OPTIMIZE once. If it does not cut the table size in half, that is a further clue that OPTIMIZE is not worth it. A 30% shrinkage after a lot of churn is somewhat typical. But it stabilizes at about that, and won't go over about 30%.
If, on the other hand, you are DELETEing most of the table all at once, then you may be better off rebuilding the table with the not-to-be-deleted rows. This combines the delete and the optimize steps into one. Then use this to flip the tables 'instantly':
RENAME TABLE real TO old, new TO real; DROP TABLE old;
And do not do any inserts during the rebuild.
When innodb_file_per_table is OFF and all data is going to be stored in ibdata files. When you remove rows, they are just marked as deleted on disk but space will be consumed by InnoDB files which can be re-used later when you insert/update more rows but it will never shrink.
If you drop some tables of delete some data then there is no any other way to reclaim that unused disk space except dump/reload method.
But, if you are using innodb_file_per_table then you can reclaim the space by running OPTIMIZE TABLE on that table. OPTIMIZE TABLE will create a new identical empty table. Then it will copy row by row data from old table to the new one. In this process a new .ibd tablespace will be created and the space will be reclaimed However, the shared tablespace-ibdata1 can still grow.
For more info- mysql reference and percona expert
I use very big innodb tables.
When I delete a data size does not shrink from the table.
What should I do for the reduction of crush syndrome delete data?
OPTIMIZE TABLE won't shrink the disk size of the table unless you haveinnodb_file_per_table=ON;
If you did not already have it ON, turning it on will only increase disk usage. This is because you will be moving that table out of ibdata1 without doing anything to shrink ibdata1.
The only way to shrink ibdata1 is to dump all tables, remove ibdata1, then reload all tables. (Yuck!)
Use OPTIMIZE TABLE yourtablenameHERE; in this case:
After doing substantial insert, update, or delete operations on an InnoDB table that has its own .ibd file because it was created with the innodb_file_per_table option enabled. The table and indexes are reorganized, and disk space can be reclaimed for use by the operating system.
In the future when in doubt just google it.
I need your advice. Need to reclaim disk space on live server with minimum downtime. We are using:
mysql -- 5.5
innodb table per file -- on
Was a huge table(70% insers/30% deletes -- it means that sometimes we delete rows from this table), this table was dropped with "drop table" command, as we expect, mysql didn't release disk space to OS, but now we need to release free space. OPTIMIZE command is possible way -- but now we don't 100% sure if we will have enough free space on disk to do this command, because previous table was huge. But in mysql documentation written:
You can run OPTIMIZE TABLE to compact or recreate a file-per-table tablespace. When you run an OPTIMIZE TABLE, InnoDB creates a new .ibd file with a temporary name, using only the space required to store actual data. When the optimization is complete, InnoDB removes the old .ibd file and replaces it with the new one.
I highlighted confusing moment in documentation, does it means, that "OPTIMIZE dropped_table;" will use only space for data that really needed, for this situation there is no actual data if we drop table early ?
If the table was created when innodb_file_per_table was OFF, then the table was written to ibdata1, and the space is not returned to the OS by DROP TABLE.
If it had been ON, then there would be a .ibd file. Did you see that file before the DROP? Does it still exist? (It should not still exist.)
When DROP removes the .ibd file, there is a slight lag (depending on the OS) while waiting for the data to be actually freed up by the OS. du and/or df reflects this lag.
OPTIMIZE TABLE will copy the table to tmpdir (I think), then drop the old table and play RENAME games. This requires extra space -- enough for a compete copy of the table. Assuming you really have a .ibd file, it will create another .ibd file (with a temp name) and shuffle files. If tmpdir points to a different "filesystem", the shuffle will involve a copy, not just a move.
If I have not explained things adequately, please provide more details -- actual commands you are proposing, size of table in question, whether the .ibd exists, etc.
I have a table which consists of heavy blobs, and I wanted to conduct some tests on it.
I know deleted space is not reclaimed by innodb, so I decided to reuse existing records by updating its own values instead of createing new records.
But I noticed, whether I delete and insert a new entry, or I do UPDATE on existing ROW, InnoDB keeps on growing.
Assuming I have 100 Rows, each Storing 500KB of information, My InnoDB size is 10MB, now when I call UPDATE on all rows (no insert/ no delete), the innodb grows by ~8MB for every run I do. All I am doing is I am storing exactly 500KB of data in each row, with little modification, and size of blob is fixed.
What can I do to prevent this?
I know about optimize table, but I cant do it because on regular usage, the table is going to be 60-100GB big, and running optimize will just stall entire server.
But I noticed, whether I delete and insert a new entry, or I do UPDATE on existing ROW, InnoDB keeps on growing.
InnoDB is a multiversion system.
This means that each time you update or delete a row, the previous version of the row gets copied to a special place called a rollback segment. The original record is rewritten with the new data (in case of an UPDATE) or marked as deleted (in case of a DELETE).
If you do a transaction rollback or execute a concurrent SELECT statement (which has to see the previous version since the new one is not yet committed), the data is retrieved from the rollback segment.
The problem with this is that InnoDB tablespace never shrinks. If the tablespace is full, new space is allocated. The space occupied by the rollback information from the commited transactions may be reused too, but if the tablespace is full it will be extended and this is permanent.
There is no supported way to shrink the ibdata* files except backing up the whole database, dropping and recreating the InnoDB files and restoring from the backup.
Even if you use innodb_file_per_table option, the rollback tablespace is still stored in the shared ibdata* files.
Note, though, that space occupied by the tables and space occupied by the tablespace are the different things. Tables can be shrunk by running OPTIMIZE and this will make them more compact (within the tablespace), but the tablespace itself cannot be shrunk.
You can reduce table size by running Optimize query
OPTIMIZE TABLE table_name;
more information can be found here