I had one table that I want drop. That table use something around 130GB. When I start drop/truncate that table, mysql server slowdown and I get a lot of issues about users using that service.
How can I drop a large table without affect mysql performance?
I already tried create a new table like that I want to drop, rename them and drop unused table (old table), but rename command hangs my mysql service.
I am using:
mysql Ver 15.1 Distrib 10.1.28-MariaDB, for Linux (x86_64) using readline 5.1
The OS is probably the problem. Some OSs take a long time to finish deleting a very large file. Sorry, but you simply have to wait.
When you DROP or TRUNCATE a table in MySQL or MariaDB and you have large buffer pool, there can be server lockup or stall of multiple seconds.The is because when a table is dropped, MySQL will perform a full scan of buffer pool looking for pages that belong to dropped table. This will lock the buffer-pool and you have to wait until it's unlock. If your buffer pool is smaller this will not effect much but if you have larger one, it becomes devastating.
One possible solution can be delete all rows before dropping a table, this will not cause a memory stall because there will be no data in memory.
Another solution can be use of non-InnoDB storage engine for temporary tables. You can specify ENGINE=MyISAM in your CREATE TABLE statement. If you have a large code base and don’t explicitly specify the storage engine, you can change the default storage engine to MyISAM by adjusting the following configuration option:
default_tmp_storage_engine = MyISAM
Of course there are some drawbacks with these solutions. You can find complete details of this bug, solution and drawbacks MySQL troubleshooting article.
The good news is that the fix itself has been implemented MySQL 8.0.23 and later.
The quickest way to empty out a large table is to create a new one with the same columns. Truncate will be a lot slower than just dropping
Disable foreign key checks
SET foreign_key_checks = 0;
Rename old table to be dropped
RENAME TABLE old TO temp;
Create the new empty table exactly like the older table
CREATE TABLE new LIKE old;
Drop old table
DROP TABLE temp
Enable foreign key checks
SET foreign_key_checks = 1;
Related
We've a problem with MySql (5.5.x), and I hope you can help us.
At some point during the day, I notice a process with state "copy to tmp table", and this query:
ALTER TABLE `MyTableName` AUTO_INCREMENT=200000001
After this, all other queries get a "Waiting for table metadata lock", and all the queries become freezed, and nothing process.
I need to Kill that process, and from that point all queries restarted.
Why? How can I fix this problem?
In MySQL 5.5, an ALTER TABLE such as the one you ran makes a copy of the whole table. The larger the table, the more time this takes. Especially if you have slow storage.
What is the size of your table (you can get this from SHOW TABLE STATUS LIKE 'MyTableName'\G and look at the data_length + index_length)?
I just did a test on my laptop. I filled a table in a MySQL 5.5 instance, until the size of the table is about 175MB. Running an alter table to set the auto-increment value takes about 5-6 seconds. Your results may be different, depending on the power of your server and the speed of storage.
While the alter table is running, the thread doing that operation holds a metadata lock on the table, which blocks all other queries, even read-only SELECT statements.
ALTER TABLE was improved in 2013, as a feature of MySQL 5.6. Some types of alters were optimized to be done "in-place" so they don't have to copy the whole table if it's not necessary. Changing the AUTO_INCREMENT is one of these operations. No matter how large the table, if you alter table to change the AUTO_INCREMENT, it's quick because it only changes an attribute of the table, without requiring copying any rows of data.
See https://dev.mysql.com/doc/refman/5.6/en/innodb-online-ddl-operations.html
In MySQL 5.5, these optimizations were not implemented. So any alter table takes a long time, proportional to the size of the table.
I would recommend the best way to fix this issue in your case is to upgrade to a newer version. MySQL 5.5 is beyond its end-of-life. Even MySQL 5.6 is reaching its end-of-life in February 2021. It's time to upgrade.
If you can't upgrade, then you should investigate what client is doing this ALTER TABLE statement. You said you noticed it at some point during the day. Track that down. In the processlist, it will tell you the client host where that SQL statement is being run from. It will also tell you the MySQL user they logged in as. You may also need to do a search on your source code of any apps or scripts that use this database. Or ask your team mates.
Once you have found the client that is doing that ALTER TABLE, try to change the time the client runs this statement to a time of day when the ALTER TABLE won't block important queries. Or ask the developer responsible if it's really necessary to do this alter table so often?
The problem could be due to a server restart, as InnoDb stores the last auto-increment index in memory and recalculates it at server restart (InnoDB AUTO_INCREMENT Counter Initialization):
If you specify an AUTO_INCREMENT column for an InnoDB table, the table handle in the InnoDB data dictionary contains a special counter called the auto-increment counter that is used in assigning new values for the column. This counter is stored only in main memory, not on disk.
To initialize an auto-increment counter after a server restart, InnoDB executes the equivalent of the following statement on the first insert into a table containing an AUTO_INCREMENT column.
SELECT MAX(ai_col) FROM table_name FOR UPDATE;
InnoDB increments the value retrieved by the statement and assigns it to the column and to the auto-increment counter for the table. By default, the value is incremented by 1. This default can be overridden by the auto_increment_increment configuration setting.
If the table is empty, InnoDB uses the value 1. This default can be overridden by the auto_increment_offset configuration setting.
Look at the mysql logs and try to find out if the server is restarting, causing the ALTER table to reset the autoincrement counter.
Try to restart the mysql server to see if you get this behaviour.
If this is the case, you could try to:
Prevent restarting the mysql server, maybe there is a cron process that restarts it once a day
Upgrade your mysql version to 8 (Autoicrement saved to table metadata):
In MySQL 8.0, this behavior is changed. The current maximum auto-increment counter value is written to the redo log each time it changes and is saved to an engine-private system table on each checkpoint.
On a server restart following a normal shutdown, InnoDB initializes the in-memory auto-increment counter using the current maximum auto-increment value stored in the data dictionary system table.
You could try to speed up "copy to tmp table" operations, skip copying to tmp table on disk mysql
References
How to make InnoDB table not reset autoincrement on server restart?
https://serverfault.com/questions/228690/mysql-auto-increment-fields-resets-by-itself
Is there a way to perform ALTER TABLE in MySQL, telling the server to skip creating a backup of the table first? I have a backup of the table already and I'm doing some tests on it (adding indexes), so I don't care if the table gets corrupted in the process. I'll just restore it from the backup. But what I do care about is for the ALTER TABLE to finish quickly, so I can see the test results.
Given that I have a big MyISAM table (700 GB) it really isn't an option to wait for couple of hours so that MySQL can first finish creating a backup of the original table, before actually adding an index to it.
It's not doing a backup; it is building the new version. (The existing table serves as a backup in case of a crash.)
With InnoDB, there are many flavors of ALTER TABLE -- some of which take essentially zero time, regardless of the size of the table. MyISAM (mostly) does the brute force way: Create an empty table with the new schema; copy all the data and build all the indexes; swap tables. For some alters, InnoDB must also do the brute force way: Example changing the PRIMARY KEY.
How should I convert MyISAM table to InnoDB on live working site in real time?
Should I just change in phpmyadmin in operations menu "Storage Engine" to InnoDB?
Would it lock all table during conversion?
You will indeed get a write lock on the table. You will still be able to read from the table while the new table is being created and copied over.
From the documentation on ALTER TABLE:
While ALTER TABLE is executing, the original table is readable by
other sessions (with the exception noted shortly). Updates and writes
to the table that begin after the ALTER TABLE operation begins are
stalled until the new table is ready, then are automatically
redirected to the new table without any failed updates.
...
The exception referred to earlier is that ALTER TABLE blocks reads
(not just writes) at the point where it is ready to install a new
version of the table .frm file, discard the old file, and clear
outdated table structures from the table and table definition caches.
At this point, it must acquire an exclusive lock. To do so, it waits
for current readers to finish, and blocks new reads (and writes).
Check out pt-online-schema-change. See example in this answer https://dba.stackexchange.com/questions/60570/best-way-to-add-a-new-column-to-a-large-table-mysql-myisam/60576#60576
It will block a table for a second or so to rename two tables.
While it might work you'd better make a copy of your table and try the storage engine change operation on the copy.
That would also give you an estimate of time needed for the operation.
Even better would be if you do that on a copy of your application and make extensive checks whether it would work at all with the new storage engine.
I have just migrated to mysql 5.5.20 and I had a performance issue with temporary tables. I have a stored procedure that creates eight of them, similar to this :
create temporary table t_opened_today
(
portfolio_id integer,
position_type_id tinyint,
open_value decimal(12,2),
today_net decimal(12,2)
);
On Mysql 5.5 it has Innodb as default storage engine. I used to be on 5.1 and that was not the case. so, it was creating the temp tables with Innodb. I verified this by looking at /tmp and didnt see any .MYI or .MYD. This was taking 0.50 second to do (or more, the execution time was bouncing all over the place), which is ridiculous.
so, I modified the table definition to include "Engine=MyISAM" and it took 0.00 seconds (as one would expect) to do nothing but create the 8 temp tables and exit.
Anyone have a clue why it would take so long in Innodb to create these temp tables? Perhaps creating temp tables using the default-storage-engine is some sort of huge no-no?
I haven't done much of anything to my.cnf as I'm just up and running. I did set increase the logs from the default.. but thats it. So the configuration is out-of-the-box.
thanks!
If you are using the default configuration then mysql will be using a single table space for all innodb tables which is probably why it is relatively slow to create your temp tables. I would suggest using MEMORY (HEAP) storage engine for your temporary tables if you have enough memory available.
I have a myisam only dedicated 32 GB RAM mysql server that is running on default configuration. I want to change the engine type to InnoDB of one table in order to avoid table locks. It has 50 million records and size on disk is around 15 GB. I am using mysql version 5.5
I guess I will need to add the following options and restart mysql.
innodb_buffer_pool_size=1G
innodb_log_file_size=100M
innodb_file_per_table=1
What else is considered to be necessary while changing the engine type?
You'll actually be running a command to convert each table.
It goes faster to first sort the table:
ALTER TABLE tablename ORDER BY primary_key_column;
Then, run the alter command:
ALTER TABLE tablename ENGINE = INNODB;
It might take a while if the table is really large, and it will use a lot of your CPU....
First of all check if your database supports InnoDB engine (I bet it is supported ;)):
SHOW ENGINES\G
If so, there is already default innodb related parameters in place, check them with:
SHOW VARIABLES LIKE '%innodb%'
and try to understand them and alter the to your specific needs. Even if you use the default params, you are now fine to play arround with InnoDB tables.
If you want to create only InnoDB tables, you can change your default storage engine, either for your current session with: SET storage_engine=INNODB; or in your config using default-storage-engine option.
By the way, the fastest way to convert a table to InnoDB is not the above described way. One can do the following to convert a table to InnoDB by simply inserting the data:
CREATE TABE new AS SELECT * FROM old WHERE 1<>1;
ALTER TABLE new ENGINE = INNODB;
INSERT INTO new SELECT * FROM old;
Of course you have to add the indexes you need manually, but its usually worth the time (and pain) you save compared to the ALTER TABLE ... on slightly bigger tables.