A large table (~10.5M rows) has been causing issues lately. I previously modified my application to use temporary tables for faster selects, but was still having issues due to UPDATE statements. Today I implemented partitions so that the writes happen more quickly, but now my temporary tables error. Its purpose is to group events, placing the first event ID of a set in the EVENT_ID column. Example: writing 4 events beginning at 1000 would result in events 1000, 1001, 1002, 1003, all with an EVENT_ID of 1000. I have tried to do away with the UPDATE statements, but that would require too much refactoring, so it is not an option. Here is the table definition:
CREATE TABLE `all_events` (
`ID` bigint NOT NULL AUTO_INCREMENT,
`EVENT_ID` bigint unsigned DEFAULT NULL,
`LAST_UPDATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`EMPLOYEE_ID` int unsigned NOT NULL,
`QUANTITY` float unsigned NOT NULL,
`OPERATORS` float unsigned NOT NULL DEFAULT '0',
`SECSEARNED` decimal(10,2) unsigned NOT NULL DEFAULT '0.00' COMMENT 'for all parts in QUANTITY',
`SECSBURNED` decimal(10,2) unsigned NOT NULL DEFAULT '0.00',
`YR` smallint unsigned NOT NULL DEFAULT (year(curdate())),
PRIMARY KEY (`ID`,`YR`),
KEY `LAST_UPDATE` (`LAST_UPDATE`),
KEY `EMPLOYEE_ID` (`EMPLOYEE_ID`),
KEY `EVENT_ID` (`EVENT_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=17464583 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
/*!50100 PARTITION BY RANGE (`YR`)
(PARTITION p2015 VALUES LESS THAN (2016) ENGINE = InnoDB,
PARTITION p2016 VALUES LESS THAN (2017) ENGINE = InnoDB,
PARTITION p2017 VALUES LESS THAN (2018) ENGINE = InnoDB,
PARTITION p2018 VALUES LESS THAN (2019) ENGINE = InnoDB,
PARTITION p2019 VALUES LESS THAN (2020) ENGINE = InnoDB,
PARTITION p2020 VALUES LESS THAN (2021) ENGINE = InnoDB,
PARTITION p2021 VALUES LESS THAN (2022) ENGINE = InnoDB,
PARTITION p2022 VALUES LESS THAN (2023) ENGINE = InnoDB,
PARTITION p2023 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */
Now in my application when running a report the statement:
CREATE TEMPORARY TABLE IF NOT EXISTS ape ENGINE=MEMORY AS
SELECT * FROM all_events
WHERE LAST_UPDATE BETWEEN '2022-05-01 00:00:00' AND CURRENT_TIMESTAMP()
Produces the error: 'Specified storage engine' is not supported for default value expressions.
Is there a way to still use temporary tables with ENGINE=MEMORY, or is there another high performance engine I can use? The statement worked until the partitioning was implemented. InnoDB is the only engine my tables can be in due to the MySQL implementation, and it has been InnoDB since before partitioning.
Edit: When removing ENGINE=MEMORY it does work, but running SHOW CREATE TABLE tells me that it's using InnoDB. I would prefer the performance increase of MEMORY vs InnoDB.
Second Edit:
The MySQL server has been crashing 2 to 3 times daily, and every time I catch it I find this error:
TRANSACTION 795211228, ACTIVE 0 sec fetching rows
mysql tables in use 13, locked 13
LOCK WAIT 866 lock struct(s), heap size 106704, 4800 row lock(s), undo log entries 1
MySQL thread id 5032986, OS thread handle 140442167994112, query id 141216988 myserver 192.168.1.100 my-user Searching rows for update
UPDATE `all_events` SET `EVENT_ID`=LAST_INSERT_ID() WHERE `EVENT_ID` IS NULL
RECORD LOCKS space id 30558 page no 16 n bits 792 index EVENT_ID of table `mydb`.`all_events` trx id 795211228 lock_mode X
It's running Galera Cluster with 3 nodes. Node 3 is the main, becomes unavailable, and 1 comes offline to resync 3. I fail over to 2 and we're usually good until it catches up, but it's causing downtime. The temp tables I'm using are for faster reads, the partitioning is my attempt at improving write performance.
Third edit:
Added example SELECT - note there are fields not in the table definition, I reduced what was displayed for simplicity of the post, but all fields in the SELECT do in fact exist.
CREATE TEMPORARY TABLE IF NOT EXISTS allpe AS
SELECT * FROM all_events
WHERE LAST_UPDATE BETWEEN ? AND ?;
CREATE TEMPORARY TABLE IF NOT EXISTS ap1 AS SELECT * FROM allpe;
CREATE TEMPORARY TABLE IF NOT EXISTS ap2 AS SELECT * FROM allpe;
SELECT PART_NUMBER, WORKCENTER_NAME, SUM(SECSEARNED) AS EARNED, SUM(SECSBURNED) AS BURNED, SUM(QUANTITY) AS QUANTITY, (
SELECT SUM(ap1.SECSEARNED)
FROM ap1
WHERE ap1.PART_NUMBER = ape.PART_NUMBER AND ap1.WORKCENTER_ID = ape.WORKCENTER_ID
) AS EARNEDALL, (
SELECT SUM(ap2.SECSBURNED)
FROM ap2
WHERE ap2.PART_NUMBER = ape.PART_NUMBER AND ap2.WORKCENTER_ID = ape.WORKCENTER_ID
) AS BURNEDALL
FROM allpe ape
WHERE EMPLOYEE_ID = ?
GROUP BY PART_NUMBER, WORKCENTER_ID, WORKCENTER_NAME, EMPLOYEE_ID
ORDER BY EARNED;
DROP TEMPORARY TABLE allpe;
DROP TEMPORARY TABLE ap1;
DROP TEMPORARY TABLE ap2;
Fourth edit:
Writing inside of stored procedure - this is not in a loop, but multiple rows can come from multiple joins to employee_presence, so I cannot get the ID and store it for writing subsequent rows.
INSERT INTO `all_events`(`EVENT_ID`,`LAST_UPDATE`,`PART_NUMBER`, `WORKCENTER_ID`,`XPPS_WC`, `EMPLOYEE_ID`,`WORKCENTER_NAME`, `QUANTITY`, `LEVEL_PART_NUMBER`,`OPERATORS`,`SECSEARNED`,`SECSBURNED`)
SELECT NULL,NOW(),NEW.PART_NUMBER,NEW.ID,OLD.XPPS_WC,ep.EMPLOYEE_ID,NEW.NAME,(NEW.PARTS_MADE-OLD.PARTS_MADE)*WorkerContrib(ep.EMPLOYEE_ID,OLD.ID),IFNULL(NEW.LEVEL_PART_NUMBER,NEW.PART_NUMBER),WorkerCount(NEW.ID)*WorkerContrib(ep.EMPLOYEE_ID,OLD.ID),WorkerContrib(ep.EMPLOYEE_ID,OLD.ID)*CreditSeconds,WorkerCount(NEW.ID)*WorkerContrib(ep.EMPLOYEE_ID,OLD.ID)*IFNULL(TIMESTAMPDIFF(SECOND, GREATEST(NEW.LAST_PART_TIME,NEW.JOB_START_TIME), now()),0)
FROM employee_presence ep WHERE ep.WORKCENTER_ID=OLD.ID;
UPDATE `all_events` SET `EVENT_ID`=LAST_INSERT_ID() WHERE `WORKCENTER_ID`=NEW.ID AND `EVENT_ID` IS NULL;
I would suppose to read the following link from dev.MySQL.com
You cannot use CREATE TEMPORARY TABLE ... LIKE to create an empty
table based on the definition of a table that resides in the mysql
tablespace, InnoDB system tablespace (innodb_system), or a general
tablespace. The tablespace definition for such a table includes a
TABLESPACE attribute that defines the tablespace where the table
resides, and the aforementioned tablespaces do not support temporary
tables. To create a temporary table based on the definition of such a
table, use this syntax instead:
CREATE TEMPORARY TABLE new_tbl SELECT * FROM orig_tbl LIMIT 0;
So it seems the correct syntax for your case will be:
CREATE TEMPORARY TABLE ape
SELECT * FROM all_events
WHERE...
In the current issue the problematic column is YR smallint unsigned NOT NULL DEFAULT (year(curdate())). This DEFAULT value is not legal for a column which is used in partitioning expression. The error will be "Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed ...".
And only when you fix this by removing the partitioning then you'll receive an error "'Specified storage engine' is not supported for default value expressions".
CREATE TABLE .. SELECT inherits main columns properties from source tables.
In the current issue the problematic column is YR smallint unsigned NOT NULL DEFAULT (year(curdate())) again. The column in temptable must inherit main properties, including DEFAULT expression - but this expression is not allowed for MEMORY engine.
As the error suggests, the expression default does not work with the MEMORY storage engine.
One solution would be to remove that default from your all_events.yr column.
The other solution is to create an empty temporary table initially as an InnoDB table, then use ALTER TABLE to remove the expression default and convert to MEMORY engine before filling it with data.
Example:
mysql> create temporary table t as select * from all_events where false;
mysql> alter table t alter column yr drop default, engine=memory;
mysql> insert into t select * from all_events;
Sufficient? If I am not mistaken, this is equivalent to what your SELECT finds (no temp tables needed):
SELECT PART_NUMBER, WORKCENTER_ID, WORKCENTER_NAME, EMPLOYEE_ID,
SUM(SECSEARNED) AS TOT_EARNED,
SUM(SECSBURNED) AS TOT_BURNED,
SUM(QUANTITY) AS TOT_QUANTITY
FROM all_events
WHERE EMPLOYEE_ID = ?
AND LAST_UPDATE >= '2022-05-01'
GROUP BY PART_NUMBER, WORKCENTER_ID, WORKCENTER_NAME;
For performance, it would need this.
INDEX(EMPLOYEE_ID, LAST_UPDATE)
Also, removing the partitioning might speed it up a little more.
else (Notes on other fixes to the path you have taken)
Since yr is not needed, avoid it by changing '*' to a list of needed columns in
CREATE TEMPORARY TABLE IF NOT EXISTS ape ENGINE=MEMORY AS
SELECT * FROM all_events
WHERE LAST_UPDATE BETWEEN '2022-05-01 00:00:00' AND CURRENT_TIMESTAMP()
WHERE ap2.PART_NUMBER = ape.PART_NUMBER AND ap2.WORKCENTER_ID = ape.WORKCENTER_ID
Add this composite index to all_events:
INDEX(PART_NUMBER, WORKCENTER_ID)
That will probably suffice to make the query fast enough without the temp tables.
Also add thatallpe` after building it.
If you are running MySQL 8.0, you can use WITH instead of needing the two extra temp tables.
As said, I'm doing and practicing a partition data in MySQL.
As I want to clear all the data, the data still there and cannot delete in one time.
This is the partition table that I created.
CREATE TABLE events
(
event_id INT unsigned NOT NULL AUTO_INCREMENT,
inserted_date DATETIME DEFAULT CURRENT_TIMESTAMP,
last_updated_date DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
data JSON NOT NULL,
KEY (event_id)
)
default charset utf8
PARTITION BY RANGE(to_days(inserted_date))
(
PARTITION p20210301 VALUES LESS THAN (to_days('2021-04-01')),
PARTITION p20210401 VALUES LESS THAN (to_days('2021-05-01')),
PARTITION p20210501 VALUES LESS THAN (to_days('2021-06-01')),
PARTITION p20210601 VALUES LESS THAN (to_days('2021-07-01')),
PARTITION p20210701 VALUES LESS THAN (to_days('2021-08-01')),
PARTITION p20210801 VALUES LESS THAN (to_days('2021-09-01')),
PARTITION p20210901 VALUES LESS THAN (to_days('2021-10-01')),
PARTITION p20211001 VALUES LESS THAN (to_days('2021-11-01')),
PARTITION p20211101 VALUES LESS THAN (to_days('2021-12-01')),
PARTITION p20211201 VALUES LESS THAN (to_days('2022-01-01')),
PARTITION p20220101 VALUES LESS THAN (to_days('2022-02-01')),
PARTITION p20220201 VALUES LESS THAN (to_days('2022-03-01')),
PARTITION p20220301 VALUES LESS THAN (to_days('2022-04-01')),
PARTITION p20220401 VALUES LESS THAN (to_days('2022-05-01')),
PARTITION p20220501 VALUES LESS THAN (to_days('2022-06-01')),
PARTITION future VALUES LESS THAN (MAXVALUE)
);
First question is everytime I select all data, different rows of data come out.
SELECT
*
FROM
plover_audit_log.events
Second I can't truncate the table. Everytime I truncate the table, the data still there.
truncate table plover_audit_log.events;
Third even I try to delete the data, it cannot delete completely. The data still there after delete.
delete from plover_audit_log.events;
I'm curious on is my MySQL software problem? Or partition table make this happen?
Because I have a backup table without partition is work well with Truncate, Delete and Select script.
data was running by one procedure, which found in processlist. So first need to kill that pid from processlist. like kill pid;
then execute truncate and check if data is still persist
Partitioned tables require a slightly different instruction to perform a TRUNCATE
To remove data from all partitions
ALTER plover_audit_log.events TRUNCATE PARTITION ALL;
To remove all data from a specific partition
ALTER plover_audit_log.events TRUNCATE PARTITION p20210301;
Reference https://dev.mysql.com/doc/refman/8.0/en/alter-table-partition-operations.html
I have a table that is empty for now but will be loaded with hundreds of millions of records. Before I do this load, I want to create some partitions on the table to improve query performance and to enable better deletion later on (just truncate an entire partition).
The alter table code I am using is:
ALTER TABLE `TABLE_NAME`
PARTITION BY RANGE (YEAR(DATE_FIELD)) (
PARTITION y1 VALUES LESS THAN (2017),
PARTITION y2 VALUES LESS THAN (2018),
PARTITION y3 VALUES LESS THAN (2019),
PARTITION ymax VALUES LESS THAN (2050)
);
When I run the code in MySQL Workbench, it executes fine without any errors. when I inspect the table, the partitions do not show up in the list:
and in the auto generated DDL, the partition is commented out:
CREATE TABLE `TABLE_NAME` (
`field1` decimal(5,2) DEFAULT NULL,
`field2` decimal(5,2) DEFAULT NULL,
`DATE_FIELD` date NOT NULL,
`field3` float DEFAULT NULL,
`field4` float DEFAULT NULL,
`field5` datetime DEFAULT NULL,
`field6` varchar(50) NOT NULL,
PRIMARY KEY (`field6`,`DATE_FIELD`),
KEY `dd_IDX1` (`DATE_FIELD`,`field1`,`field2`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (year(`DATE_FIELD`))
(PARTITION y1 VALUES LESS THAN (2017) ENGINE = InnoDB,
PARTITION y2 VALUES LESS THAN (2018) ENGINE = InnoDB,
PARTITION y3 VALUES LESS THAN (2019) ENGINE = InnoDB,
PARTITION ymax VALUES LESS THAN (2050) ENGINE = InnoDB) */
I cannot figure out why this would be. I loaded some fake records to see if the lack of data was causing the issue. I also tried commenting out the partitions and created a new table with no luck.
/*!50100 ... */ is a special type of comment. It says "If the version is 5.1.0 or later, include the text as real; else leave it as just a comment.
So, if you ran this on a 5.0 server, it would not have partitions. (5.0 did not have PARTITIONs implemented.) But 5.1 and later will.
You will see variations on this in mysqldump output.
Meanwhile, you will probably find that you gain no performance by using PARTITIONing. What were you hoping for?
After more research, I am going to chalk up the fact that the partitions do not show up in the Partitions screen when clicking Table Inspector to a bug in the GUI. When you select Alter Table and look at the partitioning tab, they show up there. Additionally, when checking the PARTITIONS information table the partitions show up there as well. See Rick James answer to understand the comment syntax.
I am using MySQL 5.6 Server. I had created a table with HASH partitiong but some how I am unable to use specific partitions in my query.
Table Structure
CREATE TABLE `testtable` (
`id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`purchased` DATE DEFAULT NULL,
KEY `ìd` (`id`),
KEY `Purchased` (`purchased`)
) ENGINE=INNODB
/*!50100 PARTITION BY RANGE ( YEAR(purchased))
SUBPARTITION BY HASH ( dayofyear(purchased))
SUBPARTITIONS 366
(PARTITION p0 VALUES LESS THAN (2015) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (2016) ENGINE = InnoDB) */
My Query
EXPLAIN PARTITIONS
SELECT *
FROM testtable
WHERE purchased BETWEEN '2014-12-29' AND '2014-12-31';
Check SQL FIDDLE Page
My EXPLAIN plan tells me that server is using all partitions instead of specific partitions.
How can I write a query so that server scans specific partitions?
And also want to know what is the problem with my current query and why it is not working?
Thanks in advance...
True. HASH partitioning is essentially useless.
Other things to note...
Having more than about 50 partitions leads to certain inefficiencies.
If you will be purging "old" rows, then consider BY RANGE and have a month in each partition. Then do the purging via DROP PARTITION. More details, including sample code: http://mysql.rjweb.org/doc.php/partitionmaint
I want to use mysql partition tables to partition a table into YEAR and the WEEK number. I know exactly how to do this with mysql merge tables but partition tables are different. Can someone please help with the following table schema?
CREATE TABLE `tableName` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`dateandtime` datetime NOT NULL,
`othervalue` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
Also does it have to be in a certain engine?
And if I store the dateandtime as a int(10) timestamp how would I do it?
CREATE TABLE `tableName` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`dateandtime` int(10) NOT NULL,
`othervalue` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
MySQL 5.1 cannot do partition by date, so you must you workaround... Usually you partition by function TO_DAYS(dateandtime), for example like this:
CREATE TABLE tbl (
... ) ENGINE=InnoDB
PARTITION BY RANGE (to_days(dateandtime)) (
PARTITION pNULL VALUES LESS THAN (0) ENGINE = InnoDB,
PARTITION p20111218 VALUES LESS THAN (TO_DAYS('2011-12-18')) ENGINE = InnoDB,
PARTITION p20111225 VALUES LESS THAN (TO_DAYS('2011-12-25')) ENGINE = InnoDB,
PARTITION pNew VALUES LESS THAN MAXVALUE ENGINE = InnoDB
)
I defined here 4 partitions - the first is just for sake of completeness, so that insert of date in future won't fail. (You can't INSERT a value for which a partition does not exist.) The first partition is for performance - NULL values will be petentionally stored there. The middle 2 partitions are actually being used, each keeping one week of data.
You can drop old partition (this is very fast compared to just DELETEing old rows) using ALTER TABLE tbl DROP PARTITION xyz. You can add new partitions by splitting the last partition:
ALTER TABLE tbl REORGANIZE PARTITION pNew INTO (
PARTITION p20120115 VALUES LESS THAN (TO_DAYS('2012-01-16')),
...
PARTITION pNew VALUES LESS THAN (MAXVALUE)
);