I tried to know how much extent ( "free space" ) does my database have after deleting a rather large table. ( Around 10GB )
I have run the command:
SELECT table_schema "Data Base Name",
round( sum( data_free ) / 1024 / 1024 / 1024 ) "Free Space in GB"
FROM information_schema.TABLES
GROUP BY table_schema;
which gave me a list of databases, and their "free spaces".
The problem is, that the database which had the 10GB table removed now has a 1500GB+ free space according to this report which is significally bigger than my actual hard drive capacity. ( which is around 200GB )
How is this possible? How could I get a more realistic report? Am I missing something?
UPDATE
As an experiment, I have added and removed an 1GB table in this database, now the report shows around 110GB more free space. Might there be a problem with my configuration, or is this a common issue?
(This is answering some of the questions buried in Comments.)
Misnomer "Free" space only includes whole blocks, not spare room inside blocks, and many other details.
Case 1: All tables are in ibdata1 -- SHOW TABLE STATUS (or the equivalent query into information_schema will show the same Data_free value, namely how much is free in ibdata1. This space can be reused by any table. It is hard to give the space back to the OS.
Case 2: All tables are file_per_table -- Now each Data_free refers to the space for the table. And the SUM() is meaningful. (ibdata1 still exists, but it does not contain any real tables; there is a lot of other stuff that InnoDB needs.)
Case 3: Mixture -- If you turn file_per_table on/off at various times, some tables will be in ibdata1, some will have their own tablespaces.
Case 4: CREATE TABLESPACE in 5.7 -- For example, you can have a tablespace for each database.
Case 5: PARTITIONed tables -- Each partition acts like a table.
Case 6: 8.0 -- Even more changes are coming.
Database == Directory In MySQL's directory tree each database can be seen as a filesystem directory. Within that directory can be seen some set of files for each table. The .frm file contains the table definition. If an .ibd file exists, the table was created with file_per_table. This may be the most reliable way to discover whether the table is file_per_table. (8.0 will have significant changes here.)
How much space can I reuse? There is no good answer. Usually inserting a row will find space in the block where it belongs, and Data_free will not shrink. But, if there were block split(s), Data_free can drop by some multiple of 16KB (the block size) or 4MB (the "extent size" - or maybe it is 8MB?). Also, random inserts lead to BTree blocks being, on average, about 69% full.
Changing innodb_file_per_table has no effect until the next CREATE TABLE or ALTER TABLE. And then it only has effect on where to put the newly created/copied data+indexes (ibdata1 or .ibd). It will not destroy data.
Big tables usually have 4MB to 7MB of Data_free. When computing how many rows you can add, don't plan on Data_free dropping below that range.
Avg_row_size should be useful. But sometimes it (and Rows) are poorly approximated. Their product (Data_length) is always correct. So, this might be a good estimate of "rows to go before grabbing more space from OS:
(Data_free - 7M) / Avg_row_size
Tablespace Recommendations: Put 'big' tables in file_per_table. Put 'tiny' tables in ibdata1 or database-specific tablespaces (5.7). Sorry, no simple recommendation on the dividing line between 'big' and 'tiny'. And it is clumsy to migrate a table: SET global innodb_file_per_table = ...;; logout; login (to pick up the global); ALTER TABLE tbl ENGINE=InnoDB;. And it is necessarily a full copy of the table.
(Caveat: I have left out many details.)
It sounds as though you do not have innondb_file_per_table set, and are therefore using a shared table space. If so, then you will be reurning the global 'allocated but unused' shared space, repeatedly for each table_schema.
Related
I have very large mysql DB(more than 100 G) and I want to migrate some changes in table. It need to posing that I have no space to make backup for this size, as results, it rolls back all changes. In clear way, when I want to alter table in mysql, it backup for it self in local disk and because of no space on my disk, it 's rolling back. Is there any one to help for this issue?
It depends on the type of ALTER change. Since MySQL 5.6, some alterations can be done "inplace" which does not use extra storage. Read https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html for details.
But many types of alterations (basically any change that affects the storage size of a row), still require a "table rebuild" which means it makes a copy of the table in the process of doing the alter. This requires extra storage, usually about the same as the size of the original table. If your server does not have enough storage space to do this, then you cannot alter the table, FULL STOP. You should have addressed this long before the table grew to this size. It is your responsibility to monitor the size of the database and make sure you have enough storage space.
You said in the comments above that it is not possible to add more storage space.
Sometimes InnoDB tablespaces can become "fragmented." That means they could store the same rows using slightly less storage space. You can accomplish defragmentation with OPTIMIZE TABLE <name>; or ALTER TABLE <name> FORCE;
A possible strategy is to optimize each of your tables, starting with the smallest table and doing one by one in order of size. You hope that you have enough space for the copy table needed to do this, and any space freed up by optimizing the small tables will help you to optimize the medium-size tables, and so on. This is a long shot, because it's also possible you don't have enough "wasted" space to recover to make a difference. You may still not be able to alter your largest tables once you're all done with this process. Unfortunately, MySQL has no way of reporting fragmentation of the tables, so you can only try it and see how much space, if any, you recover.
After that, the only other option is to delete data from the table until you can do the alteration. The alter still requires extra storage, but only as much as needed to copy the remaining rows of data. That is, if you delete 50% of the rows, then the tablespace file that stores the copy only needs roughly 50% of the storage space.
You can also try dropping some of the indexes of the table as you do your alteration. Indexes take space in the same tablespace, and it's not uncommon if you have multiple indexes for them to take more space than the data itself. If you include some DROP INDEX operations in your ALTER TABLE statement, the resulting copy will take less space. On the other hand, those indexes might be necessary for proper query optimization, and without the indexes, your application won't be usable.
I am using mysql5, and I want to shrink some 'deleted' spaces in ibd file. I already search for 'optimize table' option, but I cannot use it now, because it is a very critical table. It uses 19G in Mysql, but 33G in OS. I just figure out 33G is getting an increase.
I heard that it reuses in 33G spaces, like the black area. But why it increases every day?
MySQL does not reuse these free spaces? I mean, about 24G is never reused?
Thanks for the read, I hope not my poor English makes you are confused when you read it.
The option you are looking for is innodb_use_trim=1, in combination with innodb_file_per_table=1, if the version of MySQL you are using is new enough to have it. This will release back empty pages by punching holes in the tablespace files.
The valid parts of a table are scattered throughout the .ibd. file. The Operating System does not have a way to reuse freed up pieces from the middle of a table.
So, InnoDB never shrinks a tablespace, only grows it.
Meanwhile, new INSERTs will fill in some of the freed space.
What led to this situation? Did you DELETE lots of rows? If you expect to do another big DELETE, see the following advice for next time: http://mysql.rjweb.org/doc.php/deletebig
Meanwhile, if the table was created with innodb_file_per_table = 1, then OPTIMIZE TABLE will copy the table over and release the old copy to the OS. But... You need extra disk space to do the task. And it will block some operations against the table. (The details depend on which version of MySQL/MariaDB you are using.)
Why do you need to recoup the space? If you are not running low on disk space, then does it really matter?
Low on space?
Show us SHOW CREATE TABLE. There may be some tips on making the table smaller. (This does not solve the problem, but delays future problems.) For example, if an INT (4 bytes) can be changed to SMALLINT (2 bytes), that could help shrink the table. Or an INDEX might be redundant; DROPping it would help.
The ALTER to change such things might do the OPTIMIZE at the same time. (In MySQL 5.0, virtually all ALTERs are performed by copying the table over.)
Another tip when disk space is terribly tight... (Assuming innodb_file_per_table has always been on): OPTIMIZE smaller tables; Some of them may shrink a little bit, thereby giving you some extra disk space.
Another tip:
Determine which tables are in their own tablespaces.
Find out how much free space is in ibdata1.
SET innodb_file_per_table=OFF;
ALTER TABLE t ENGINE=InnoDB; for some of the small tables is step 1. But stop before Data_free of ibdata1 drops too small.
That tip will free up the space for those table that could be moved without causing ibdata1 to grow.
I have a rather heavy database
I have to change server, so I export the database with a dump, then I retrieved it on my new server
But after importing it, the size of the database is different :
Old server : 772 Mo
New server : 414 Mo
The difference is enormous, I have to worry? (Do you think there are missing things?)
Is not it better to do an export then an import manually?
You didn't mention if these size numbers from from the filesystem (du -ch, etc), or from a query. I'm guessing they are from the filesystem. As long as you didn't get import errors, your data is probably OK.
Check for FRAGMENTATION in your SOURCE tables. That is likely why your source is larger than your target. Basically, when rows are updated, it may no longer fit in the same data block, and that data block is split into two, leaving some free space in two blocks (16KB used on disk, but 9KB used in the data file).
Check for fragmented tables:
select ENGINE, TABLE_NAME,Round( DATA_LENGTH/1024/1024) as data_length ,
round(INDEX_LENGTH/1024/1024) as index_length,
round(DATA_FREE/ 1024/1024) as data_free
from information_schema.tables
where DATA_FREE > 0;
Check for total disk used and total free space:
select sum((DATA_LENGTH + INDEX_LENGTH)/1024/1024) as TTL_MB_USED ,
sum(DATA_FREE)/1024/1024 as TTL_MB_FREE
from information_schema.tables
where table_schema='<your schema>';
That may help account for the difference in size between source and target.
I found this answer to be excellent to describe Fragmentation: https://serverfault.com/a/265885
first at all you must understand that Mysql tables get fragmented when a row is updated, so it's a normal situation. When a table is created, lets say imported using a dump with data, all rows are stored with no fragmentation in many fixed size pages. When you update a variable length row, the page containing this row is divided in two or more pages to store the changes, and these new two (or more) pages contains blank spaces filling the unused space.
I am trying to reduce the disk space usage of a table in an RDS instance of MySQL 5.6.23. It's an InnoDB table with about 8 million rows and 30 columns. Several of the columns are of type TEXT NULL DEFAULT NULL. One of the reasons why the table is so big is because rather than deleting rows from this table, they are instead marked as deleted via a flag column named 'deleted'.
After reading the MySQL documentation on storage requirements:
http://dev.mysql.com/doc/refman/5.7/en/storage-requirements.html
It seems as though the storage required for a TEXT field depends on the length of text in the field rather than being a fixed size (L + 2 bytes, where L < 2^16 and where L is the length of the value in bytes). So although I've read elsewhere that these fields are in fact fixed width, I processed about 50,000 rows marked as deleted and set all their TEXT column values to null.
However, there was no reduction in disk space reported either by the MySQL client or the AWS Console RDS interface. Why didn't this free up disk space?
When you set the column value to NULL InnoDB would have to reorganize the record storage in order to reduce the total amount of disk space used by the table. You should see a reduction if you a dummy ALTER TABLE that is not dummy enough for MySQL to notice a short-circuit way to do it making it actually rebuild the table, or manually drop, re-create, and reinsert the records. OPTIMIZE TABLE should do it as well.
Sasha's answer may or may not apply.
After setting the column to NULL, any freed blocks are made available for future INSERTs / UPDATEs. But the freed blocks are not given back to the OS. Whether a block is freed depends on a lot of details.
The amount of disk space for a TEXT field depends both on the amount of text and the Row_format ("Compact", etc). A TEXT column may be entirely or partially stored in a block separate from the rest of the data.
If your table was created while innodb_file_per_table was ON, then OPTIMIZE TABLE will give the free space back to the OS. And SHOW TABLE STATUS will show some decrease in values.
If innodb_file_per_table had been OFF, freed up space is left in ibdata1, but that file is not shrunken. It can be shrunken only by dump all tables; stop mysqld; remove ibdata1; restart; reload. (Yuck.) OPTIMIZE TABLE will increase Data_free inside ibdata1.
(Assuming OFF) This will make the table more manageable, but leave a lot of free space in an un-shrunken ibdata1:
SET innodb_file_per_table = ON;
ALTER TABLE foo ENGINE=InnoDB;
If you anticipate growth in ibdata1 for other reasons; this may be wise to do. Otherwise, it just makes the disk space problem worse.
I'm using InnoDB database with a single file configuration (in /var), so no innodb_file_per_table.
In the MySql workbench, when I query for the databases used space, with this query
SELECT table_schema "Database", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB"
FROM information_schema.TABLES GROUP BY table_schema;
It says that I have 47 GB of data.
However the size of ibdata1 is 99 GB...
I know that ibdata1 contains a bunch of other things other that table data, like Table Indexes, MVCC (Multiversioning Concurrency Control) Data and Table Metadata
So my question is: Is it normal that supposedly 52 GB of ibdata1 is medatada and a bunch of other things? Usually, how much data beside table data should the ibdata1 file contains?
No, it is not normal that you would have that much metadata. It is normal though that the ibdata file can grow to a ridiculous size if you aren't using innodb_file_per_table.
Your ibdata file will grow when your database grows, but it will never actually shrink.
So, for example, if you had 130 GB of data at one point and deleted a bunch of it, the ibdata file would still be 130 GB after the data was purged. It will just have a bunch of "free space" that it will then use for subsequent inserts.
As for shrinking the file, there's not much you can really do aside from wiping out your database and restoring it. This answer has some good instructions on how to do that.
Howto: Clean a mysql InnoDB storage engine?
You also might want to consider using innodb_file_per_table as deleting data from a table and later optimizing that table will actually shrink the size of the individual table files
There are a few reasons for having a bunch of "extra" space in ibdata1, but the most likely cases are:
You have deleted large amounts of data in the past. When you delete rows or drop tables, although free space will be made available in the file, the file itself will never shrink.
You may have an excessive amount of undo log space (or have at some point in the past). Undo logs are kept during DELETE and UPDATE operations, and for very long-running operations touching many rows can grow quite large. Again, if the file is expanded to hold this data it will never shrink.
As previously mentioned using innodb_file_per_table can help with this if you expect to regularly drop tables and want to get the disk space back. My blog post The basics of InnoDB space file layout may help you understand what is included in the ibdata1 file.