I have just migrated to mysql 5.5.20 and I had a performance issue with temporary tables. I have a stored procedure that creates eight of them, similar to this :
create temporary table t_opened_today
(
portfolio_id integer,
position_type_id tinyint,
open_value decimal(12,2),
today_net decimal(12,2)
);
On Mysql 5.5 it has Innodb as default storage engine. I used to be on 5.1 and that was not the case. so, it was creating the temp tables with Innodb. I verified this by looking at /tmp and didnt see any .MYI or .MYD. This was taking 0.50 second to do (or more, the execution time was bouncing all over the place), which is ridiculous.
so, I modified the table definition to include "Engine=MyISAM" and it took 0.00 seconds (as one would expect) to do nothing but create the 8 temp tables and exit.
Anyone have a clue why it would take so long in Innodb to create these temp tables? Perhaps creating temp tables using the default-storage-engine is some sort of huge no-no?
I haven't done much of anything to my.cnf as I'm just up and running. I did set increase the logs from the default.. but thats it. So the configuration is out-of-the-box.
thanks!
If you are using the default configuration then mysql will be using a single table space for all innodb tables which is probably why it is relatively slow to create your temp tables. I would suggest using MEMORY (HEAP) storage engine for your temporary tables if you have enough memory available.
Related
I had one table that I want drop. That table use something around 130GB. When I start drop/truncate that table, mysql server slowdown and I get a lot of issues about users using that service.
How can I drop a large table without affect mysql performance?
I already tried create a new table like that I want to drop, rename them and drop unused table (old table), but rename command hangs my mysql service.
I am using:
mysql Ver 15.1 Distrib 10.1.28-MariaDB, for Linux (x86_64) using readline 5.1
The OS is probably the problem. Some OSs take a long time to finish deleting a very large file. Sorry, but you simply have to wait.
When you DROP or TRUNCATE a table in MySQL or MariaDB and you have large buffer pool, there can be server lockup or stall of multiple seconds.The is because when a table is dropped, MySQL will perform a full scan of buffer pool looking for pages that belong to dropped table. This will lock the buffer-pool and you have to wait until it's unlock. If your buffer pool is smaller this will not effect much but if you have larger one, it becomes devastating.
One possible solution can be delete all rows before dropping a table, this will not cause a memory stall because there will be no data in memory.
Another solution can be use of non-InnoDB storage engine for temporary tables. You can specify ENGINE=MyISAM in your CREATE TABLE statement. If you have a large code base and don’t explicitly specify the storage engine, you can change the default storage engine to MyISAM by adjusting the following configuration option:
default_tmp_storage_engine = MyISAM
Of course there are some drawbacks with these solutions. You can find complete details of this bug, solution and drawbacks MySQL troubleshooting article.
The good news is that the fix itself has been implemented MySQL 8.0.23 and later.
The quickest way to empty out a large table is to create a new one with the same columns. Truncate will be a lot slower than just dropping
Disable foreign key checks
SET foreign_key_checks = 0;
Rename old table to be dropped
RENAME TABLE old TO temp;
Create the new empty table exactly like the older table
CREATE TABLE new LIKE old;
Drop old table
DROP TABLE temp
Enable foreign key checks
SET foreign_key_checks = 1;
I am using MySQL Server-5.1.58 Community log. The problem is after deleting the data the allocated space of MySQL database is not getting free and as a result day by day the backup size of my using database is increasing.
Please kindly let me know any tool which can resolve the issue.
Remember that MySQL locks the table during the time OPTIMIZE TABLE is running
For your MySQL version from the official documentation:
OPTIMIZE TABLE should be used if you have deleted a large part of a
table or if you have made many changes to a table with variable-length
rows (tables that have VARCHAR, VARBINARY, BLOB, or TEXT columns).
Deleted rows are maintained in a linked list and subsequent INSERT
operations reuse old row positions. You can use OPTIMIZE TABLE to
reclaim the unused space and to defragment the data file
Additional notes for InnoDB:
For InnoDB tables, OPTIMIZE TABLE is mapped to ALTER TABLE, which
rebuilds the table to update index statistics and free unused space in
the clustered index. Beginning with MySQL 5.1.27, this is displayed in
the output of OPTIMIZE TABLE when you run it on an InnoDB table, as
shown here:
mysql> OPTIMIZE TABLE foo;
Table does not support optimize, doing recreate + analyze instead
So:
OPTIMIZE [NO_WRITE_TO_BINLOG | LOCAL] TABLE
tbl_name [, tbl_name] ...
By default, OPTIMIZE TABLE statements are written to the binary log so
that they will be replicated to replication slaves. Logging can be
suppressed with the optional NO_WRITE_TO_BINLOG keyword or its alias
LOCAL.
I have a myisam only dedicated 32 GB RAM mysql server that is running on default configuration. I want to change the engine type to InnoDB of one table in order to avoid table locks. It has 50 million records and size on disk is around 15 GB. I am using mysql version 5.5
I guess I will need to add the following options and restart mysql.
innodb_buffer_pool_size=1G
innodb_log_file_size=100M
innodb_file_per_table=1
What else is considered to be necessary while changing the engine type?
You'll actually be running a command to convert each table.
It goes faster to first sort the table:
ALTER TABLE tablename ORDER BY primary_key_column;
Then, run the alter command:
ALTER TABLE tablename ENGINE = INNODB;
It might take a while if the table is really large, and it will use a lot of your CPU....
First of all check if your database supports InnoDB engine (I bet it is supported ;)):
SHOW ENGINES\G
If so, there is already default innodb related parameters in place, check them with:
SHOW VARIABLES LIKE '%innodb%'
and try to understand them and alter the to your specific needs. Even if you use the default params, you are now fine to play arround with InnoDB tables.
If you want to create only InnoDB tables, you can change your default storage engine, either for your current session with: SET storage_engine=INNODB; or in your config using default-storage-engine option.
By the way, the fastest way to convert a table to InnoDB is not the above described way. One can do the following to convert a table to InnoDB by simply inserting the data:
CREATE TABE new AS SELECT * FROM old WHERE 1<>1;
ALTER TABLE new ENGINE = INNODB;
INSERT INTO new SELECT * FROM old;
Of course you have to add the indexes you need manually, but its usually worth the time (and pain) you save compared to the ALTER TABLE ... on slightly bigger tables.
Ran into an interesting problem with a MySQL table I was building as a temporary table for reporting purposes.
I found that if I didn't specify a storage engine, the DROP TEMPORARY TABLE command would hang for up to half a second.
If I defined my table as ENGINE = MEMORY this short hang would disappear.
As I have a solution to this problem (using MEMORY tables), my question is why would a temporary table take a long time to drop? Do they not use the MEMORY engine by default? It's not even a very big table, a couple of hundred rows with my current test data.
Temporary tables, by default, will be created where ever the mysql configuration tells it to, typically /tmp or somewhere else on a disk. You can set this location (and even multiple locations) to a RAM disk location such as /dev/shm.
Hope this helps!
If the temporary file is created with InnoDb engine, which may be the case if your default engine was InnoDb, and the InnoDb buffer pool is large, DROP TEMPORARY TABLE may take some time since it needs to scan all pages to discard.
It was mentionned in a comment to this stack overflow question.
Note also that DROP (TEMPORARY) TABLE uses a LOCK that may have huge impact on all your server. See for example this.
At my work, we recently had a server slow down because we had an InnoDb buffer pool of 80 Gb and some SQL requests had been optimized using InnoDb temporary tables.
About 100 such DROP TEMPORARY TABLE requests every 5 minutes were sufficient to have a huge impact. And the problem was hard to debug since slow query log would tell us that UPDATEs of a single row accessed by primary key in some other table was taking two seconds, and there was an enormous amount of such updates. But even if most query time was spent on these updates, the problem was really because of the DROP TEMPORARY TABLE requests.
Hi I am using Mysql 5.0.x
I have just changed a lot of the tables from MyISAM to InnoDB
With the MyISAM tables it took about 1 minute to install our database
With the InnoDB it takes about 15 minute to install the same database
Why does the InnoDB take so long?
What can I do to speed things up?
The Database install does the following steps
1) Drops the schema
2) Create the schema
3) Create tables
4) Create stored procedures
5) Insert default data
6) Insert data via stored procedure
EDIT:
The Inserting of default data takes most of the time
Modify the Insert Data step to start a transaction at the start and to commit it at the end. You will get an improvement, I guarantee it. (If you have a lot of data, you might want to break the transaction up to per table.)
If you application does not use transactions at all, then you should set the paramater innodb_flush_log_at_trx_commit to 2. This will give you a lot of performance back because you will almost certainly have auto_commit enabled and this generates a lot more transactions than InnoDB's default parameters are configured for. This setting stops it unnecessarily flushing the disk buffers on every commit.
15 minutes doesn't seem excessive to me. After all, it's a one-time cost.
I'm not certain, but I would imagine that part of the explanation is the referential integrity isn't free. InnoDB has to do more work to guarantee it, so of course it would take up more time.
Maybe your script needs to be altered to add constraints after the tables are created.
Like duffymo said, disable your constraints(indexes and foreing/primary keys) before inserting the data.
Maybe you should restore some indexes before the data inserted via stored procedure, if its use a lot of select statements