What's is better to use in a stored procedure: a temporary table or a memory table?
The table is used to stored summary data for reports.
Are there any trade offs that developers should be aware off?
CREATE TEMPORARY TABLE t (avg (double));
or
CREATE TABLE t (avg (double)) ENGINE=MEMORY;
Why is this restricted to just the two options? You can do:
CREATE TEMPORARY TABLE t (avg double) ENGINE=MEMORY;
Which works, although I'm not sure how to check if the memory engine is actually being used here.
Of the two, I'd use a temporary table for report.
A memory table holds data across user sessions & connections, so you'd have to truncate it every time to make sure you wouldn't be using data from someone else. Assuming you put in whats necessary to maintain a memory table depending on your needs, it's fine - the temp table is a little safer from a maintenance perspective.
A temporary table will only exist for the duration of your session. A table declared with Engine=Memory will persist across user sessions / connections but will only exist in the lifetime of the MySQL instance. So if MySQL gets restarted the table goes away.
In MySQL, temporary tables are seriously crippled:
http://dev.mysql.com/doc/refman/5.6/en/temporary-table-problems.html
You cannot refer to a TEMPORARY table more than once in the same query.
For example, the following does not work:
mysql> SELECT * FROM temp_table, temp_table AS t2;
ERROR 1137: Can't reopen table: 'temp_table'
I Just wanted to point out that, in 2021 using MariaDB-10.3.27, the code #biziclop said doesn't work, is not the case any more, this is possible:
CREATE TEMPORARY TABLE tmp1 AS
SELECT * FROM products LIMIT 10;
SELECT * FROM tmp1, tmp1 AS t2;
(I just tested it)
Related
I have a need to "update" some table data I receive from external source (every time I receive "all" data, with some fields for some records updated).
There's no unique field or combination of fields, and thus I figured the best way would be to every time to wipe out all data from DB and write all (now updated) data in again. There are up to a 1000 records (there will never be more than that), about 15 short fields each: text, numbers, datetime. And I'm writing it to remote DB (so, it's slow).
Currently I'm doing:
delete from `table` where `date_dt` > ?
and then for each row
INSERT INTO `table` ( `field_0`,`field_1`,... ) VALUES (?,?,...)
It's not only slow, but it's possible that the end user may not see the complete data while I'm still inserting.
I figured I could do:
CREATE TEMPORARY TABLE `temp_table` ( ... ); -- same structure as in main table
INSERT INTO `temp_table` ( `field_0`,`field_1`,... ) VALUES (?,?,...) -- repeat 1000x
START TRANSACTION;
DELETE FROM `table`;
INSERT INTO `table` SELECT * FROM `temp_table`;
DROP `temp_table`;
COMMIT;
Does this makes any sense? What's is a better way of solving this?
The speed of filling up the temp table with data is not crucial, but filling the main table with data is (so users don't see incomplete data, or the period of time they do is minimal).
mysqlimport --delete will truncate the table first, and then load your external data from a CSV file. It runs many times faster than doing INSERT one row at a time.
See https://dev.mysql.com/doc/refman/5.7/en/mysqlimport.html
I did a presentation in April 2017 about performance of bulk data loads for MySQL:
https://www.slideshare.net/billkarwin/load-data-fast
P.S.: Don't use the temp table solution if you have a MySQL replication environment. This is a well-known way of breaking replication. If the slave restarts in between your creation of the temp table and the INSERT...SELECT that reads from the temp table, then the slave will find the temp table is gone, and this will result in an error and stop replication. This might seem unlikely, but it does happen eventually.
I create a temporary table variable (#table). Then I inner join it with another table. It takes a long time to display result. I try again with #table. It's normal. What's wrong with it?
If you store too much data in temp table or temp table variable then it result in poor performance. Because these temp tables variables not allow indexing and in temp table most of the time developer forget to use proper indexing therefore during join complete table scan happen which slow the query output.
Another important point to notice, avoid joins on varchar column.
I am executing a query
ALTER TABLE message ADD COLUMN syncid int(10) NOT NULL DEFAULT 0;
MySQL returned error:
ERROR 1878 (HY000): Temporary file write failure.
message table info:
engine type:InnoDB
rows:15786772
index length:1006.89 MB
data length:11.25 GB
How to fix it?
MySQL implements ALTER TABLE as a table re-creation, so two copies of the table exists on the system at some stage during the process. You will need over 12 GB free space for this operation.
Free some space. Alternatively, set your server to use a different temporary directory, where there is enough space.
Alternative to the alternative (the WHILE might need to be wrapped in a stored procedure):
create a new table (temp_table) with the new structure
transfer data in small batches from original_table into temp_table
drop original_table and rename temp_table
-- useful only if concurrent access is allowed during migration
LOCK TABLES original_table WRITE, temp_table WRITE;
SELECT COUNT(*) INTO #anythingleft FROM original_table;
WHILE #anythingleft DO
-- transfer data
INSERT INTO temp_table
SELECT
original_table.old_stuff,
"new stuff"
FROM original_table
ORDER BY any_sortable_column_with_unique_constraint -- very important!
LIMIT 1000; -- batch size, adjust to your situation
DELETE FROM original_table
ORDER BY any_sortable_column_with_unique_constraint
LIMIT 1000; -- ORDER BY and LIMIT clauses MUST be exactly the same as above
SELECT COUNT(*) INTO #anythingleft FROM original_table;
END WHILE;
-- delete, rename
DROP TABLE original_table;
UNLOCK TABLES;
RENAME TABLE old_table TO original_table;
If your table uses InnoDB, a more elaborate solution is possible with SELECT ... FOR UPDATE; instead of table locks, but I trust you get the idea.
Sorry for the late answer or digging up this old topic, but the following tools can help you with that:
pt-online-schema-change
github/gh-ost
Both tools recreate the table in the fashion that #RandomSeed proposed, but in a simpler way.
However please ensure that there is enough space on the file system. Those tools don't need more space in the temporary folder, which is interesting when you're mounting your temporary folder on a separate drive / RAMdisk.
I was reading about tables using MEMORY ENGINE (tables stored in ram).
Is it possible to use CREATE TABLE AS SELECT syntax, but have the created table us MEMORY ENGINE?
Is there a limit to the size of a MEMORY table? Could create a MEMORY table that is a copy of a pysical 1.5 GB table?
Yes, it can be done, but not with simple syntax CREATE TABLE t AS SELECT ... - you will need to specify table columns in full statement, Sample:
CREATE TABLE t (col1 INT(11), col2 INT(11))
ENGINE=MEMORY
AS
SELECT * FROM another_t
Maximum size of table by default if 16Mb, but it can be adjusted with max_heap_table_size server system variable. But please, note, that this limit is per engine - not per table. I.e. all your memory tables will share it. To restrict size of individual table, you'll need to operate on session value for max_heap_table_size, like
mysql> SET max_heap_table_size = 24*1024*1024;
you can also create a temporary table this way:
CREATE TEMPORARY TABLE IF NOT EXISTS tmp ENGINE=MEMORY AS (
SELECT 'abc' As tomcat, 'def' As apache);
Or this way:
CREATE TEMPORARY TABLE IF NOT EXISTS tmp ENGINE=MEMORY AS (
SELECT tomcat, apache From servers);
Be advised, the first option will create the two columns as varchar(3), meanwhile the second option will import any column information from the original table, even their comments.
If two users create two TEMPORARY tables at the same time in my mysql database using a PHP script, will it create two different tables? Can those users use their own table without facing any trouble and will those table be automatically deleted?
thanks :)
CREATE TEMPORARY TABLE TempTable ( ID int, Name char(100) ) TYPE=HEAP;
INSERT INTO TempTable VALUES( 1, "Foo bar" );
SELECT * FROM TempTable;
DROP TABLE TempTable;
As stated in the manual:
A TEMPORARY table is visible only to the current connection, and is dropped automatically when the connection is closed. This means that two different connections can use the same temporary table name without conflicting with each other or with an existing non-TEMPORARY table of the same name. (The existing table is hidden until the temporary table is dropped.)
Temporary tables are created in a current session; so, two users can create two temp. tables at the same time in their threads. These tables will be removed on session closing.
You can use [IF NOT EXISTS] in this query...
reference