I tried to increase my disk space by deleting the some old records, but my disk space is reduced at the time of running delete query. So I cancelled that query execution.. but still the disk space is reduced.
What i need to do ? is any other way ?
Deleting records does not automatically regain you disk space, as the disk space allocated to the database files stays the same. You in fact may increase the size of the transaction log!
If you have deleted a bunch of records it may be possible to regain some disk space.
First you'll need to determine how much disk space is actually used & how much is available.
The following script should tell you this (taken from DBA Exchange ):
SELECT RTRIM(name) AS [Segment Name], fileid AS [File Id], groupid AS [Group Id], filename AS [File Name],
CAST(size/128.0 AS DECIMAL(10,2)) AS [Size in MB],
CAST(FILEPROPERTY(name, 'SpaceUsed')/128.0 AS DECIMAL(10,2)) AS [Space Used],
CAST(size/128.0-(FILEPROPERTY(name, 'SpaceUsed')/128.0) AS DECIMAL(10,2)) AS [Available Space],
CAST((CAST(FILEPROPERTY(name, 'SpaceUsed')/128.0 AS DECIMAL(10,2))/CAST(size/128.0 AS DECIMAL(10,2)))*100 AS DECIMAL(10,2)) AS [Percent Used]
FROM sysfiles
ORDER BY groupid DESC
You can then use the command DBCC SHRINKFILE to shrink the files. - eg
DBCC SHRINKFILE(1, 240000)
Would shrink the file with fileID = 1 to 240 GB.
Note however -
Shrinking database files is a bad idea in general, as you will cause indexes to become fragmented, and hence cause performance problems.
If you do resort to shrinking the file - make sure there is a reasonable amount of free space left after the shrink - probably at least 15%
If at all possible do not shrink the database file. Shrinking the transaction log file is less of a problem (ensure it is truncated first so there is space to shrink the file).
Read the Technet article here about managing the transaction log http://technet.microsoft.com/en-us/library/ms345382(v=sql.105).aspx
Maybe consider getting more storage space for your database...
Related
I have a database named "bongoTv" where lots of table but I found one table its size about 20GB with less amount of data.
After removing few row storage did not reduced. Then I ran a command
OPTIMIZE TABLE notifiation to re-indexing. But It increase its size to 25GB.
As per my undersetting with other DBMS it should be reduce its size but why its size increased, I think it cached previous information somewhere.
After searching on web I found need to configure with innodb_file_per_table=ON. But here in my configuration it is also enabled. But it did not worked.
Need expert opinion who dedicatedly working on this MySQL.
In that case what need to do from my end, what is the solution this issue?
#Louis &
#P.Salmon Can you help me on this?
Thanks in Advance who is going to help me on this.
In general, InnoDB tablespace files never shrink. If you delete data, it makes some space "unused" and over time InnoDB will try to reuse unused space before expanding the tablespace file further.
But there is also tablespace fragmentation. As you delete rows and leave small gaps of unused space, those small gaps may not be usable for new data. So over time, the gaps grow in number, and the tablespace uses more space than it should, if you were to store the same data as compactly as possible.
The free space that comprise full extents, or contiguous 1MB areas, are shown as data_free when you run SHOW TABLE STATUS. But smaller gaps of unused space are not shown. MySQL has no way of reporting the "crumbs" of unused space.
When you use OPTIMIZE TABLE on an InnoDB table, it still cannot shrink the tablespace, it only copies data to a new tablespace. It tries to defragment the data, leaving out the gaps where possible. So if there are a lot of large and small gaps in your old tablespace, the new tablespace should have a smaller total size.
However, while filling pages of the new tablespace, InnoDB deliberately leaves 1/16 of each page unused, to allow for future updates that might need just a little bit more room. So in theory, you might see OPTIMIZE TABLE cause the file to grow larger if the original was very compact and the new file was created with more "elbow room."
But that still does not account for the 20GB to 25GB change you saw. That might be because sizes are cached. That is, the old file was in fact 25GB, but the table status was not reporting it. MySQL 8.0 especially has some caching behavior on some table statistics: https://bugs.mysql.com/bug.php?id=86170
So how to reduce the table size in MySQL?
Deleting rows is the most effective way. If you don't need data to be in the database anymore, delete it. If you might need data for archival purposes but don't need to query it every day, then copy it out to some long-term archiving format, or another database instance on a large-capacity server, and then delete the data from your primary database.
Changing data types to be smaller. For example, why use a BIGINT (64-bits) when a SMALLINT (16-bits) is sufficient for the values you store? It may seem like a small change, but it adds up. Values are stored in the row, but also stored again in any indexes that include that column.
Using compression. The best results are in text and strings that store readable text. The amount of compression depends on the nature of the data. Don't count on this too much, because at best one can expect a 2:1 ratio of compression, and often not even that much.
Ultimately, databases tend to grow larger, and often even the rate of growth accelerates. If you accumulate a lot of data and never delete or archive them, you must make a strategy to support the growth. You may just have to get larger and larger storage volumes.
I've learned that execute DELETE sql in mysql will not release space, so i do an experiment about it.
During the experiment, i find when my data size is large(about 20000 rows), i delete half of the rows, the data length is not reduced.
But when my data size is about 1000 rows, i try to delete half of the rows. Unexpectedly the data length will be reduced.
It's really confusing. I have search a lot about this problem, but got nothing. What's the reason?
My MySQL version is 5.6
InnoDB table statistics are not updated every time you DELETE. So the data_length, index_length, rows, average_row_length and data_free may not describe the precise state of the table at all times.
You might like to read:
https://www.percona.com/blog/2011/10/06/when-does-innodb-update-table-statistics-and-when-it-can-bite/
https://www.percona.com/blog/2017/09/11/updating-innodb-table-statistics-manually/
Even after the table stats are updated, they are not exact. They are estimates based on a limited sample of the table.
It should also be mentioned that the size of a tablespace file remains at its high-water mark, even if the data_length is reduced. The difference is space that is part of the file on your filesystem, but it is not occupied by data or indexes anymore. InnoDB should reuse that "free" space within the tablespace before it expands the file further.
I tried to know how much extent ( "free space" ) does my database have after deleting a rather large table. ( Around 10GB )
I have run the command:
SELECT table_schema "Data Base Name",
round( sum( data_free ) / 1024 / 1024 / 1024 ) "Free Space in GB"
FROM information_schema.TABLES
GROUP BY table_schema;
which gave me a list of databases, and their "free spaces".
The problem is, that the database which had the 10GB table removed now has a 1500GB+ free space according to this report which is significally bigger than my actual hard drive capacity. ( which is around 200GB )
How is this possible? How could I get a more realistic report? Am I missing something?
UPDATE
As an experiment, I have added and removed an 1GB table in this database, now the report shows around 110GB more free space. Might there be a problem with my configuration, or is this a common issue?
(This is answering some of the questions buried in Comments.)
Misnomer "Free" space only includes whole blocks, not spare room inside blocks, and many other details.
Case 1: All tables are in ibdata1 -- SHOW TABLE STATUS (or the equivalent query into information_schema will show the same Data_free value, namely how much is free in ibdata1. This space can be reused by any table. It is hard to give the space back to the OS.
Case 2: All tables are file_per_table -- Now each Data_free refers to the space for the table. And the SUM() is meaningful. (ibdata1 still exists, but it does not contain any real tables; there is a lot of other stuff that InnoDB needs.)
Case 3: Mixture -- If you turn file_per_table on/off at various times, some tables will be in ibdata1, some will have their own tablespaces.
Case 4: CREATE TABLESPACE in 5.7 -- For example, you can have a tablespace for each database.
Case 5: PARTITIONed tables -- Each partition acts like a table.
Case 6: 8.0 -- Even more changes are coming.
Database == Directory In MySQL's directory tree each database can be seen as a filesystem directory. Within that directory can be seen some set of files for each table. The .frm file contains the table definition. If an .ibd file exists, the table was created with file_per_table. This may be the most reliable way to discover whether the table is file_per_table. (8.0 will have significant changes here.)
How much space can I reuse? There is no good answer. Usually inserting a row will find space in the block where it belongs, and Data_free will not shrink. But, if there were block split(s), Data_free can drop by some multiple of 16KB (the block size) or 4MB (the "extent size" - or maybe it is 8MB?). Also, random inserts lead to BTree blocks being, on average, about 69% full.
Changing innodb_file_per_table has no effect until the next CREATE TABLE or ALTER TABLE. And then it only has effect on where to put the newly created/copied data+indexes (ibdata1 or .ibd). It will not destroy data.
Big tables usually have 4MB to 7MB of Data_free. When computing how many rows you can add, don't plan on Data_free dropping below that range.
Avg_row_size should be useful. But sometimes it (and Rows) are poorly approximated. Their product (Data_length) is always correct. So, this might be a good estimate of "rows to go before grabbing more space from OS:
(Data_free - 7M) / Avg_row_size
Tablespace Recommendations: Put 'big' tables in file_per_table. Put 'tiny' tables in ibdata1 or database-specific tablespaces (5.7). Sorry, no simple recommendation on the dividing line between 'big' and 'tiny'. And it is clumsy to migrate a table: SET global innodb_file_per_table = ...;; logout; login (to pick up the global); ALTER TABLE tbl ENGINE=InnoDB;. And it is necessarily a full copy of the table.
(Caveat: I have left out many details.)
It sounds as though you do not have innondb_file_per_table set, and are therefore using a shared table space. If so, then you will be reurning the global 'allocated but unused' shared space, repeatedly for each table_schema.
While looking to solve the Disk Space Issue, I have been through several posts and found that Shrinking the database is one option while some others are Deleting Error Log files and transaction log files. But,
I found the first option controversial where in order to free up the
unused space SQL uses an ugly process and results in Index
fragmentation that affects performance in the long run. Meaning that
after deallocating the space we are giving authority to the Operating
system do what it needs to with it.
So, would it be fair enough to remove the shrink option from the list of the available options?
This is true that shrinking a database is not recommended. You can understand it like this when you shrink the database then it leads to increase in fragmentation now to reduce the fragmentation you try to rebuilt the index which will eventually lead to increase in your database size.
You can run the query to test it by yourself
SELECT name, (size*8) Size_KB
FROM sys.database_files
GO
-- Check Fragmentations in the database
SELECT avg_fragmentation_in_percent, fragment_count
FROM sys.dm_db_index_physical_stats (DB_ID(), OBJECT_ID('SecondTable'), NULL, NULL, 'LIMITED')
GO
-- Shrink the Database
DBCC SHRINKDATABASE (ShrinkIsBed);
GO
-- Name of the Database and Size
SELECT name, (size*8) Size_KB
FROM sys.database_files
GO
-- Check Fragmentations in the database
SELECT avg_fragmentation_in_percent, fragment_count
FROM sys.dm_db_index_physical_stats (DB_ID(), OBJECT_ID('SecondTable'), NULL, NULL, 'LIMITED')
GO
-- Rebuild Index on FirstTable
ALTER INDEX IX_SecondTable_ID ON SecondTable REORGANIZE
GO
-- Name of the Database and Size
SELECT name, (size*8) Size_KB
FROM sys.database_files
GO
-- Check Fragmentations in the database
SELECT avg_fragmentation_in_percent, fragment_count
FROM sys.dm_db_index_physical_stats (DB_ID(), OBJECT_ID('SecondTable'), NULL, NULL, 'LIMITED')
GO
Query source from here
You can check Don’t Touch that Shrink Database Button!
What Happens when you Shrink a Database?
When you click that shrink database button (or leave a DB in
autoshrink, or schedule a job to perform shrinks), you are asking SQL
Server to remove the unused space from your database’s files.The
process SQL uses is ugly and results in Index fragmentation that
affects performance in the long run. You’ve deallocated that space and
are letting the O/S do what it needs to with it. If you have a growing
database (as the majority of production databases tend to be), this
means that that database will grow again. Depending on your autogrowth
settings (another pet peeve for another post) this growth will
probably be more than necessary and you will end up shrinking again…
At best this is just extra work (shrink grow/shrink grow) and the
resulting file fragmentation is handled alright by your I/O subsystem.
At worse this is causing that index fragmentation I mentioned, file
fragmentation, interrupting what would have otherwise been contiguous
files and potentially causing I/O related performance problems. Really
though, you are wasting time and introducing index fragmentation.
I have just turned on page compression on a table (SQL 2008 Ent) using the following command:
ALTER TABLE [dbo].[Table1] REBUILD PARTITION = ALL
WITH
(DATA_COMPRESSION = PAGE
)
The hard drive now contains 50GB less space than before. I'm guessing that I need to run a command to reclaim the space. Anyone know it?
I feel embarrassed even asking this question, but is it something that could be fixed by shrink the database in question? As it compressed the pages, perhaps it left the space free all throughout the file, and the data files just need to be condensed and shrunk to reclaim the space...
If it created a new, compressed copy of the table and then removed the old one from the file, but didn't shrink the file internally, this might also explain your sudden lack of space on the drive as well.
If this is the case, then a simple "DBCC SHRINKDATABASE('my_database')" should do the trick. NOTE: This may take a long time, and lock the database during that time so as to prevent access, so schedule it wisely.
Have you checked using the table size using sp_spaceused?
Disk space used does not equal space used by data. The compression will have affected log file size (all has to be logged) and required some free working space (like the rule of thumb that index rebuild requires free space = 1.2 times largest table space).
Another option is that you need to rebuild the clustered index because it's fragmented. This compacts data and is the only way to reclaim space for text columns.
Also, read Linchi Shea's articles on data compression