Using: MySQL 5.6 on Debian 9, total DB size is around 450Gb
Updated to 5.7, ran mysql_upgrade, noticed that around 150 GB has been taken up. 2 tables are really large and they stayed in 'copying to tmp table' for a couple of hours
Noticed innodb_file_per_table was on and created large ibd files that weren't there previously.
Restored from a snapshot, disabled the file_per_table, ran mysql_upgrade again. 100GB gone, which is almost 1/4 of my total DB.
In the first case, it pulled the data from ibdata and put it into a separate file but ibdata never shrinks so taken space almost doubled.
What happens in the second case? Does the temp table get created within the ibdata file that never shrinks so even when table is not used anymore - space is still gone?
Another thing I noticed is that space consumption doesn't start until query has been in copying to tmp table status for like an hour or so.
1) Is there any way to avoid/minimize space increase?
Would running update with file_per_table on, then disabling it and running alter table engine innodb free up the space?
2) Any way to predict how much space will be occupied? At least per table
3) How does max_tmp_table_size play into this?
It sounds like you painted yourself into a corner by not running innodb_file_per_table from the start, so now you have a huge, unshrinkable ibdata1 file.
1) There isn't.
1.1) It might reduce the overall space usage by rebuilding the tables outside the ibdata1 file, then rebuild them again to inside ibdata1, reusing some of the unused space inside ibdata1
2) Yes:
SELECT TABLE_SCHEMA, TABLE_NAME, DATA_LENGTH, INDEX_LENGTH FROM information_schema.TABLES;
3) It doesn't. The tables you are seeing are probably tables being rebuilt for some reason (not sure why, I have to admit I haven't seen that happen from mysql_upgrade before). max_tmp_table_size is only for implicit (when a query plan says using temporary) and explicity (CREATE TEMPORARY TABLE ...) temporary tables, not for table rebuilds.
The only(?) way to switch to file_per_table without the disk bloat is
Dump the data.
Get a fresh install (or otherwise get rid of ibdata1).
Reload (with file_per_table on).
Related
I am using mysql5, and I want to shrink some 'deleted' spaces in ibd file. I already search for 'optimize table' option, but I cannot use it now, because it is a very critical table. It uses 19G in Mysql, but 33G in OS. I just figure out 33G is getting an increase.
I heard that it reuses in 33G spaces, like the black area. But why it increases every day?
MySQL does not reuse these free spaces? I mean, about 24G is never reused?
Thanks for the read, I hope not my poor English makes you are confused when you read it.
The option you are looking for is innodb_use_trim=1, in combination with innodb_file_per_table=1, if the version of MySQL you are using is new enough to have it. This will release back empty pages by punching holes in the tablespace files.
The valid parts of a table are scattered throughout the .ibd. file. The Operating System does not have a way to reuse freed up pieces from the middle of a table.
So, InnoDB never shrinks a tablespace, only grows it.
Meanwhile, new INSERTs will fill in some of the freed space.
What led to this situation? Did you DELETE lots of rows? If you expect to do another big DELETE, see the following advice for next time: http://mysql.rjweb.org/doc.php/deletebig
Meanwhile, if the table was created with innodb_file_per_table = 1, then OPTIMIZE TABLE will copy the table over and release the old copy to the OS. But... You need extra disk space to do the task. And it will block some operations against the table. (The details depend on which version of MySQL/MariaDB you are using.)
Why do you need to recoup the space? If you are not running low on disk space, then does it really matter?
Low on space?
Show us SHOW CREATE TABLE. There may be some tips on making the table smaller. (This does not solve the problem, but delays future problems.) For example, if an INT (4 bytes) can be changed to SMALLINT (2 bytes), that could help shrink the table. Or an INDEX might be redundant; DROPping it would help.
The ALTER to change such things might do the OPTIMIZE at the same time. (In MySQL 5.0, virtually all ALTERs are performed by copying the table over.)
Another tip when disk space is terribly tight... (Assuming innodb_file_per_table has always been on): OPTIMIZE smaller tables; Some of them may shrink a little bit, thereby giving you some extra disk space.
Another tip:
Determine which tables are in their own tablespaces.
Find out how much free space is in ibdata1.
SET innodb_file_per_table=OFF;
ALTER TABLE t ENGINE=InnoDB; for some of the small tables is step 1. But stop before Data_free of ibdata1 drops too small.
That tip will free up the space for those table that could be moved without causing ibdata1 to grow.
I deleted thousands of thousands of rows in a table, but the free space in (C:) decreased from 40 Gb to 15 Gb. What happened? Wasn't it supposed to go up?
I've already restarted the MySQL Server and even my computer, but the problem remains. I'm using Windows 10 and MySQL 8.0
Removing rows is not a guaranteed way to reclaim free disk space. MySQL tends to store data for tables in a single file, or one file per disk. Generally it just marks rows in this file as deleted, so if you insert new rows after, they can take up this space.
To reclaim disk space, MySQL would have to completely rewrite these data files as it's not really possible to (cheaply) delete some data from the middle of a large file.
One reason I can think of for actually having less space, is that you might have the binlog enabled, but it's hard to know for sure what took this extra space without knowing which file sizes increased.
One way to really reclaim this disk space is to rebuild the entire table from scratch.
I need your advice. Need to reclaim disk space on live server with minimum downtime. We are using:
mysql -- 5.5
innodb table per file -- on
Was a huge table(70% insers/30% deletes -- it means that sometimes we delete rows from this table), this table was dropped with "drop table" command, as we expect, mysql didn't release disk space to OS, but now we need to release free space. OPTIMIZE command is possible way -- but now we don't 100% sure if we will have enough free space on disk to do this command, because previous table was huge. But in mysql documentation written:
You can run OPTIMIZE TABLE to compact or recreate a file-per-table tablespace. When you run an OPTIMIZE TABLE, InnoDB creates a new .ibd file with a temporary name, using only the space required to store actual data. When the optimization is complete, InnoDB removes the old .ibd file and replaces it with the new one.
I highlighted confusing moment in documentation, does it means, that "OPTIMIZE dropped_table;" will use only space for data that really needed, for this situation there is no actual data if we drop table early ?
If the table was created when innodb_file_per_table was OFF, then the table was written to ibdata1, and the space is not returned to the OS by DROP TABLE.
If it had been ON, then there would be a .ibd file. Did you see that file before the DROP? Does it still exist? (It should not still exist.)
When DROP removes the .ibd file, there is a slight lag (depending on the OS) while waiting for the data to be actually freed up by the OS. du and/or df reflects this lag.
OPTIMIZE TABLE will copy the table to tmpdir (I think), then drop the old table and play RENAME games. This requires extra space -- enough for a compete copy of the table. Assuming you really have a .ibd file, it will create another .ibd file (with a temp name) and shuffle files. If tmpdir points to a different "filesystem", the shuffle will involve a copy, not just a move.
If I have not explained things adequately, please provide more details -- actual commands you are proposing, size of table in question, whether the .ibd exists, etc.
I have a 20gb MySQL database which contains about 30 million rows of data in one table. I wanted to delete a column within that table to clear up some space.
I dropped the column via phpMyAdmin which took about 15 minutes and checked the disk usage in my server. It went from 20gb to 28gb used after deleting a column that was filled with data.
Did I do something wrong? Am I missing something obvious?
I did nothing else while I was trying to delete the column.
Update: Actually, when I go to the database summary page that lists the tables in phpMyAdmin, it also lists their size. The size of the tables only adds up to 10gb.
When I check the size being used on my server via df -h it says /dev/vda is using 28gb.
The only other files on the server are a few small PHP files. What is taking up this 18gb gap?
Most likely explanation is that you are not using file-per-table mode, and the ibdata1 file that contains all tables and indexes increased during the ALTER TABLE. When you delete a column, MySQL has to create a new copy of the table, move all the data into it (minus the column you deleted) and then drop the original table. Near the end of this process, two copies of the table (minus the column you deleted in one copy) need to be stored simultaneously. Then when that's complete, it drops the original copy and that space can be reused in the future.
Unfortunately, there's no way to shrink the ibdata1. It remains at a size equal to its high-water mark. To recover the disk space, you have to:
Dump all InnoDB tables (or convert them to MyISAM temporarily)
Shut down MySQL Server
Remove ibdata1 and ib_logfile*
Set innodb_file_per_table=1 in your my.cnf
Restart. The missing ibdata1 will be created at the initial default size.
Import your data.
Now the tables will occupy a separate file per table. When you ALTER or OPTIMIZE them, they will recover space. This mode is enabled by default in MySQL 5.6.
I'm using InnoDB database with a single file configuration (in /var), so no innodb_file_per_table.
In the MySql workbench, when I query for the databases used space, with this query
SELECT table_schema "Database", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB"
FROM information_schema.TABLES GROUP BY table_schema;
It says that I have 47 GB of data.
However the size of ibdata1 is 99 GB...
I know that ibdata1 contains a bunch of other things other that table data, like Table Indexes, MVCC (Multiversioning Concurrency Control) Data and Table Metadata
So my question is: Is it normal that supposedly 52 GB of ibdata1 is medatada and a bunch of other things? Usually, how much data beside table data should the ibdata1 file contains?
No, it is not normal that you would have that much metadata. It is normal though that the ibdata file can grow to a ridiculous size if you aren't using innodb_file_per_table.
Your ibdata file will grow when your database grows, but it will never actually shrink.
So, for example, if you had 130 GB of data at one point and deleted a bunch of it, the ibdata file would still be 130 GB after the data was purged. It will just have a bunch of "free space" that it will then use for subsequent inserts.
As for shrinking the file, there's not much you can really do aside from wiping out your database and restoring it. This answer has some good instructions on how to do that.
Howto: Clean a mysql InnoDB storage engine?
You also might want to consider using innodb_file_per_table as deleting data from a table and later optimizing that table will actually shrink the size of the individual table files
There are a few reasons for having a bunch of "extra" space in ibdata1, but the most likely cases are:
You have deleted large amounts of data in the past. When you delete rows or drop tables, although free space will be made available in the file, the file itself will never shrink.
You may have an excessive amount of undo log space (or have at some point in the past). Undo logs are kept during DELETE and UPDATE operations, and for very long-running operations touching many rows can grow quite large. Again, if the file is expanded to hold this data it will never shrink.
As previously mentioned using innodb_file_per_table can help with this if you expect to regularly drop tables and want to get the disk space back. My blog post The basics of InnoDB space file layout may help you understand what is included in the ibdata1 file.