MyISAM files size has increased (doubled) - mysql

One of my table's file size (a .MYD file) has increased by about 100%.
If I look into the data there is just a normal daily load during the last few days. What could have caused this file size increase?
myisamchk
As suggested by a user I tried sudo myisamchk -r tablename. The result of this operation was "Fixing index 1" "Fixing index 2" ... and the exact same file size.
Edit: After running the command a second time the file size decreased to the normal (half) size.

#e4c5 has the most likely answer. He is asking you to remove the deleted rows. MySQL will not reclaim the space from deleted rows. As more and more rows are deleted, the larger your .MYD becomes. The only way to reclaim the space is to rebuild the .MYD - either by recreating the table, using the optimize table command or running myisamchk -r.
The following command will rebuild the table.
optimize table $table_name;
The myisamchk -r command line approach works too - but be sure the table is locked to prevent users from accessing it (otherwise you run the very real risk of losing data). Be sure to flush table before using it again after running myisamchk repair or MySQL will detect the table has changed and run an unnecessary check against it. Of the three methods, myisamchk -r is the most prone to data lost (but still my preferred method- you just have to be careful and prepared for the worst).

It sounds like you are replacing the entire table every day. Here's an optimal way to do it; no bloat, no OPTIMIZE:
CREATE TABLE new LIKE real;
populate `new`
RENAME TABLE real TO old,
new TO real;
DROP TABLE old;
That way, you build a clean, new, table and swap it in. Note that the table is always available; that is, no downtime.

Based on #rick-james's answer I looked further and found another thread. This does the trick... very fast and no downtime (if the table is not written during the run).
-- table_org is the name of the table to 'optimize'
CREATE TABLE `table_new` LIKE `table_org`;
-- disable keys for quick inserts
ALTER TABLE `table_new` DISABLE KEYS;
INSERT INTO `table_new`
SELECT * FROM `table_org`;
ALTER TABLE `table_new` ENABLE KEYS;
RENAME TABLE `table_org` TO `table_old`, `table_new` TO `table_org`;
DROP TABLE `table_old`;
-- is not needed, but just to have an up-to-date data
ANALYZE TABLE `table_org`;

Related

Is there a way to turn off the creation of a temp table during ALTER TABLE?

Is there a way to perform ALTER TABLE in MySQL, telling the server to skip creating a backup of the table first? I have a backup of the table already and I'm doing some tests on it (adding indexes), so I don't care if the table gets corrupted in the process. I'll just restore it from the backup. But what I do care about is for the ALTER TABLE to finish quickly, so I can see the test results.
Given that I have a big MyISAM table (700 GB) it really isn't an option to wait for couple of hours so that MySQL can first finish creating a backup of the original table, before actually adding an index to it.
It's not doing a backup; it is building the new version. (The existing table serves as a backup in case of a crash.)
With InnoDB, there are many flavors of ALTER TABLE -- some of which take essentially zero time, regardless of the size of the table. MyISAM (mostly) does the brute force way: Create an empty table with the new schema; copy all the data and build all the indexes; swap tables. For some alters, InnoDB must also do the brute force way: Example changing the PRIMARY KEY.

Optimizing table results in "Waiting for table metadata lock"

Running OPTIMIZE TABLE results in "Waiting for table metadata lock". Checking SHOW PROCESSLIST confirms optimizing is the only active query.
I have a table that is 750GB, and 69GB left on the drive. To free up space I decided to cleanup that table. I've turned all access to that server off, and started by deleting old records, which would have ended up taking forever. It has been decided that the table can just be truncated but a small chunk of the data needed to be extracted first. Problem, even a simple SELECT * FROM my_table LIMIT 1 takes hours before it is manually killed. Is this an indexing issue? And if so is, 69GB enough for the index process.
If you have something else you can delete to free up disk space, that might release the lock.
Kill the process, make sure there are no tmp tables left around clogging disk.
Then do the cleanup a different way...
CREATE TABLE new LIKE real;
DROP any indexes you don't immediately need from `new`.
INSERT INTO new
SELECT ... FROM real
WHERE ...;
RENAME TABLE real TO old, new TO real;
DROP TABLE old;
If you make it this far, ADD back the indexes you should have.
Potential problem: If the table is Engine=InnoDB and was created with innodb_file_per_table=OFF, then this is not sufficient to free up any disk space.
If you don't delete more than 90% of the table, and you have only 69GB of free space, the process will eventually fail.
"For the index process" -- This phrase "does not compute".
OPTIMIZE TABLE does:
Create a new table like the old one, but perhaps without any indexes other than the PRIMARY KEY.
Copy all the rows (other than deleted ones) over.
Build the indexes (assuming they were not incrementally built in step 2).
RENAME and DROP (as above)

Mysql : Is adding column take same time as renaming column [duplicate]

I have a 12 GB table full of pictures, I'm trying to rename the blob column that holds the data, and it is taking forever. Can someone give me a blow by blow account of why it is taking so long to rename the column? I would have thought that this operation would be pretty quick, no matter the size of the table?
EDIT: The query I ran is as follows
alter table `rails_production`.`pictures` change `data` `image_file_data` mediumblob NULL
It appears that most of the time is spent waiting for mysql to make a temporary copy of the pictures table, which since it is very large is taking a while to do.
It is on the list of things to do, to change the picture storage from the database to the filesystem.
EDIT2: Mysql Server version: 5.0.51a-24+lenny2 (Debian)
I can't give you the blow-by-blow (feature request #34354 would help, except that it probably wouldn't be back-ported to MySQL 5.0), but the extra time is due to the fact that an ALTER ... CHANGE may change the type of the column (and column attributes, if any), which necessitates converting the values stored in the column and other checks. MySQL 5.0 doesn't include optimizations for when the new type and attributes are the same as the old. From the documentation for ALTER under MySQL 5.0:
In most cases, ALTER TABLE works by making a temporary copy of the original table. The alteration is performed on the copy, and then the original table is deleted and the new one is renamed. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table are stalled until the new table is ready, and then are automatically redirected to the new table without any failed updates.
[...]
If you use any option to ALTER TABLE other than RENAME, MySQL always creates a temporary table, even if the data wouldn't strictly need to be copied (such as when you change the name of a column).
Under 5.1, ALTER has some additional optimizations:
In some cases, no temporary table is necessary:
Alterations that modify only table metadata and not table data can be made immediately by altering the table's .frm file and not touching table contents. The following changes are fast alterations that can be made this way:
Renaming a column, except for the InnoDB storage engine.
[...]
Because MySQL will rebuild the entire table when you make schema changes.
This is done because it's the only way of doing it in some cases, and it makes it much easier for the server to rebuild it anyway.
Yes mysql does a temporary copy of the table. I don't think there's an easy way around that. You should really think about to store the pictures on the filesystem and only store paths in mysql. That's the only way to fasten it up, I guess.

How to dump a single table in MySQL without locking?

When I run the following command, the output only consists of the create syntax for 'mytable', but none of the data:
mysqldump --single-transaction -umylogin -p mydatabase mytable > dump.sql
If I drop --single-transaction, I get an error as I can't lock the tables.
If I drop 'mytable' (and do the DB), it looks like it's creating the INSERT statements, but the entire DB is huge.
Is there a way I can dump the table -- schema & data -- without having to lock the table?
(I also tried INTO OUTFILE, but lacked access for that as well.)
The answer might depend on the database engine that you are using for your tables. InnoDB has some support for non-locking backups. Given your comments about permissions, you might lack the permissions required for that.
The best option that comes to mind would be to create a duplicate table without the indicies. Copy all of the data from the table you would like to backup over to the new table. If you create your query in a way that can easily page through the data, you can adjust the duration of the locks. I have found that 10,000 rows per iteration is usually pretty darn quick. Once you have this query, you can just keep running it until all rows are copied.
Now that you have a duplicate, you can either drop it, truncate it, or keep it around and try to update it with the latest data and leverage it as a backup source.
Jacob

Why does it take so long to rename a column in mysql?

I have a 12 GB table full of pictures, I'm trying to rename the blob column that holds the data, and it is taking forever. Can someone give me a blow by blow account of why it is taking so long to rename the column? I would have thought that this operation would be pretty quick, no matter the size of the table?
EDIT: The query I ran is as follows
alter table `rails_production`.`pictures` change `data` `image_file_data` mediumblob NULL
It appears that most of the time is spent waiting for mysql to make a temporary copy of the pictures table, which since it is very large is taking a while to do.
It is on the list of things to do, to change the picture storage from the database to the filesystem.
EDIT2: Mysql Server version: 5.0.51a-24+lenny2 (Debian)
I can't give you the blow-by-blow (feature request #34354 would help, except that it probably wouldn't be back-ported to MySQL 5.0), but the extra time is due to the fact that an ALTER ... CHANGE may change the type of the column (and column attributes, if any), which necessitates converting the values stored in the column and other checks. MySQL 5.0 doesn't include optimizations for when the new type and attributes are the same as the old. From the documentation for ALTER under MySQL 5.0:
In most cases, ALTER TABLE works by making a temporary copy of the original table. The alteration is performed on the copy, and then the original table is deleted and the new one is renamed. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table are stalled until the new table is ready, and then are automatically redirected to the new table without any failed updates.
[...]
If you use any option to ALTER TABLE other than RENAME, MySQL always creates a temporary table, even if the data wouldn't strictly need to be copied (such as when you change the name of a column).
Under 5.1, ALTER has some additional optimizations:
In some cases, no temporary table is necessary:
Alterations that modify only table metadata and not table data can be made immediately by altering the table's .frm file and not touching table contents. The following changes are fast alterations that can be made this way:
Renaming a column, except for the InnoDB storage engine.
[...]
Because MySQL will rebuild the entire table when you make schema changes.
This is done because it's the only way of doing it in some cases, and it makes it much easier for the server to rebuild it anyway.
Yes mysql does a temporary copy of the table. I don't think there's an easy way around that. You should really think about to store the pictures on the filesystem and only store paths in mysql. That's the only way to fasten it up, I guess.