partitions and sub partitions - mysql

The following CREATE TABLE statement to partition a table works as expected, without error.
CREATE TABLE `ox_data_archive_20120108` (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY LIST(to_days(datetime )) (
PARTITION `1Jan10` VALUES IN (to_days('2010-01-01') ),
PARTITION `2Jan10` VALUES IN (to_days('2010-01-02') ),
PARTITION `3Jan10` VALUES IN (to_days('2010-01-03') )
);
What I need to do is to create subpartitions based on date + zoneid. I tried the following:
CREATE TABLE mypart (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY LIST(to_days(datetime ))
SUBPARTITION BY KEY(zoneid) (
PARTITION `1Jan10` VALUES IN (to_days('2010-01-01') )
(Subpartition s1, Subpartition s2),
PARTITION `2Jan10` VALUES IN (to_days('2010-01-02') )
(Subpartition s3, Subpartition s4),
PARTITION `3Jan10` VALUES IN (to_days('2010-01-03') )
(Subpartition s5, Subpartition s6)
);
Inserting into this table:
INSERT INTO mypart VALUES (1, 2, 3, '2012-01-31 04:10:03');
results in the following error:
ERROR 1526 (HY000): Table has no partition for value 734898
My query expects to use the zoneid subpartition based on dates. Is it possible?

Contrary to your assertion that the first table works without error, inserting the sample data into it:
INSERT INTO `ox_data_archive_20120108` VALUES (1, 2, 3, '2012-01-31 04:10:03');
results in the same error as for the second table. The value given in the error (734898) happens to be the value for to_days('2012-01-31'). You get this error because you only have partitions for January 1st through 3rd, 2010. Both the day-of-month and the year for the sample data are outside the defined partitions. Instead of TO_DAYS (which returns the number of days from year 0 to the given date), you probably want DAYOFMONTH. Since each partition is contiguous, a RANGE partition seems more appropriate than a LIST.
Off topic, you only need to specify separate subpartition definitions when you want to set options for the subpartitions. Since you're not doing that, a SUBPARTITIONS 2 clause will do the same thing as your statement, but is simpler.
CREATE TABLE mypart (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY RANGE(DAYOFMONTH(`datetime`))
SUBPARTITION BY KEY(zoneid)
SUBPARTITIONS 2 (
PARTITION `01` VALUES LESS THAN 2, -- Note: 0 is valid day-of-month
PARTITION `02` VALUES LESS THAN 3,
PARTITION `03` VALUES LESS THAN 4,
...
);

Related

How to create partition mysql 8

I am trying to create a table with partition with following way
create table transaction_demo
(
trx_id bigint not null primary key,
created_datetime datetime
) partition by range (YEAR(created_datetime)) (
partition p1 values less than (2019),
partition p2 values less than (2020),
partition p3 values less than (2021)
);
But it is not working. I am getting following error
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
How to fix this?
Then define primary key for each column which will be part of the partition.
CREATE TABLE transaction_demo (
trx_id int NOT NULL,
created_datetime datetime NOT NULL,
other_column_1 int NULL,
other_column_2 varchar(222) NULL,
PRIMARY KEY(trx_id,created_datetime)
)
PARTITION BY RANGE( YEAR(created_datetime) )(
PARTITION p1 VALUES LESS THAN (2019),
PARTITION p2 VALUES LESS THAN (2020),
PARTITION p3 VALUES LESS THAN (2021)
);

Correctly optimising MySQL data for date range queries?

I have a table with lots of numeric data. I need to query this to get the closest row for a specific date_added and name.
My problem is that this data is not ordered by date, so when returning results I need to include ORDER BY date_added (or it doesn't return the correct row). Currently doing this takes a good 90 seconds to run because of this ORDER BY condition.
Are there any ways I can further optimise this? I've already indexed the date_added and name columns, so I'm not really sure what else can be done. I considered creating a new table with the data reordered in date_added order, but this isn't practical as new entries need to be added regularly.
I've stored the numeric data as decimal as it can potentially be very small, very large or both. Perhaps storing this data in a different way would be more efficient?
Add a compound index on name and date. The query above will run without using filesort.
An alternative way for the query:
SELECT date_added, data_1, data_2, data_3, data_4, data_5, data_6, data_7, data_8, data_9
FROM numeric_data
WHERE date_added =
(select min(date_added) from numeric_data where date_added >= '2018-05-03 11:00:00' and name = 'aaa')
and name = 'aaa'
limit 1;
Fiddle: http://sqlfiddle.com/#!9/4e8d89/1 .
You can use range partitioning:
https://dev.mysql.com/doc/refman/5.7/en/partitioning-range.html
You need to define your partitions depends on the date range you have.
CREATE TABLE `numeric_data` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`date_added` datetime NOT NULL,
`name` varchar(8) COLLATE utf8mb4_unicode_ci NOT NULL,
`data_1` decimal(30,17) NOT NULL,
`data_2` decimal(30,17) NOT NULL,
`data_3` decimal(30,17) NOT NULL,
`data_4` decimal(30,17) NOT NULL,
`data_5` decimal(30,17) NOT NULL,
`data_6` decimal(30,17) NOT NULL,
`data_7` decimal(30,17) NOT NULL,
`data_8` decimal(30,17) NOT NULL,
`data_9` decimal(30,17) NOT NULL,
PRIMARY KEY (`id`),
KEY `date_added` (`date_added`),
KEY `name` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=60000000 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE( TO_DAYS(date_added) ) (
PARTITION p1 VALUES LESS THAN (TO_DAYS('2018-01-01')),
PARTITION p2 VALUES LESS THAN (TO_DAYS('2018-02-01')),
PARTITION p3 VALUES LESS THAN (TO_DAYS('2018-03-01')),
PARTITION p4 VALUES LESS THAN (TO_DAYS('2018-04-01')),
PARTITION future VALUES LESS THAN MAXVALUE
);
For the below query will only use partition "future" :
SELECT date_added, data_1, data_2, data_3, data_4, data_5, data_6, data_7, data_8, data_9
FROM numeric_data
WHERE date_added >= '2018-05-03 11:00:00'
AND name = 'aaa'
ORDER BY date_added LIMIT 1

MySQL Partitioning Error

I'm running MySQL 5.6 and have the following table:
CREATE TABLE `rawdata` (
`RawDataSeq` int(10) unsigned NOT NULL AUTO_INCREMENT,
`SpeciesSeq` int(10) unsigned DEFAULT NULL,
`DataSourceSeq` int(10) unsigned DEFAULT NULL,
`MeasurementSeq` int(10) unsigned DEFAULT NULL,
`RecordingDateTime` datetime DEFAULT NULL,
`RawSensorData` varchar(100) DEFAULT NULL,
`RawDataNumeric` decimal(36,18) DEFAULT NULL,
`RawDataString` varchar(100) DEFAULT NULL,
`RawDataDate` datetime DEFAULT NULL,
`RawDataTime` varchar(45) DEFAULT NULL,
`RawDataDateTime` datetime DEFAULT NULL,
`ActiveRecord` int(10) unsigned DEFAULT NULL,
`LastUpdateDate` datetime DEFAULT NULL,
`UserId` varchar(6) DEFAULT NULL,
`BadData` varchar(200) DEFAULT NULL,
PRIMARY KEY (`RawDataSeq`)
) ENGINE=MyISAM AUTO_INCREMENT=5676258 DEFAULT CHARSET=latin1
I'm trying to partition the table by 'RecordingDateTime' using the following command:
ALTER TABLE rawdata PARTITION BY RANGE (RecordingDateTime) (
PARTITION p0 VALUES LESS THAN (2013),
PARTITION p1 VALUES LESS THAN (2014),
PARTITION p2 VALUES LESS THAN (2015),
PARTITION p3 VALUES LESS THAN (UNIX_TIMESTAMP('2015-06-01 00:00:00')),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
I'm receiving the following error message:
Query: ALTER TABLE rawdata PARTITION BY RANGE (RecordingDateTime) ( PARTITION p0 VALUES LESS THAN (2013), PARTITION p1 VALUES LESS THAN...
Error Code: 1659
Field 'RecordingDateTime' is of a not allowed type for this type of partitioning
Any help is greatly appreciated.
Change to
PRIMARY KEY (`RawDataSeq`, RecordingDateTime)
ALTER TABLE rawdata PARTITION BY RANGE (TO_DAYS(RecordingDateTime)) (
PARTITION p_old VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION p2013 VALUES LESS THAN (TO_DAYS('2014-01-01')),
PARTITION p2014 VALUES LESS THAN (TO_DAYS('2015-01-01')),
PARTITION p2015 VALUES LESS THAN (TO_DAYS('2016-01-01')),
PARTITION future VALUES LESS THAN MAXVALUE
);
The code you need each December is found here.
I suspect there is a problem with the datetime value and the integer values you are using. One method is:
ALTER TABLE rawdata PARTITION BY RANGE (year(RecordingDateTime)) (
PARTITION p0 VALUES LESS THAN (2013),
PARTITION p1 VALUES LESS THAN (2014),
PARTITION p2 VALUES LESS THAN (2015),
PARTITION p3 VALUES LESS THAN MAXVALUE
);
If you don't store the RecordingDateTime as a timestamp, you can also try:
ALTER TABLE rawdata PARTITION BY RANGE (year(RecordingDateTime)*100 + month(RecordDateTime) (
PARTITION p0 VALUES LESS THAN (201300),
PARTITION p1 VALUES LESS THAN (201400),
PARTITION p2 VALUES LESS THAN (201500),
PARTITION p3 VALUES LESS THAN (201506),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
Otherwise, store the value as a timestamp and use timestamp ranges.

MySQL table partitioning on timestamp

I have partitioned a table (because of an out of memory error - table got too big). I have partitioned it on a timestamp column as shown below:
CREATE TABLE test (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( unix_timestamp(dob) ) (
PARTITION p2012 VALUES LESS THAN (unix_timestamp('2013-01-01 00:00:00')),
PARTITION p2013 VALUES LESS THAN (unix_timestamp('2014-01-01 00:00:00')),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
I was hoping that the process of partitioning would also help in speeding up a couple of my queries whihc take a few hours to run; however, this type of partitioning doesn't seem to kick in and all partitions are still being used and scanned through for the queries. I have tried, and failed, with a couple more approaches:
1) Tried to use different range for the partitioning
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
However, that results in an error: Error Code: 1486. Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
2) Gave up on changing partitioning to be recognized by the query optimizer, and as suggested in MySQL's Doc - 18.5 Partition Selection tried specifying which partitions to use in the select statement instead:
select * from t2 partition (p2012)
But, that returns a syntax error Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(p2012) LIMIT 0, 1000' at line 1
Does anybody have any suggestions what else I could try to utilize table partitioning to optimize the queries?
You can use UNIX_TIMESTAMP() function. Example from MySQL docs:
CREATE TABLE quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-07-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-10-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-01-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-04-01 00:00:00') ),
PARTITION p6 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-07-01 00:00:00') ),
PARTITION p7 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-10-01 00:00:00') ),
PARTITION p8 VALUES LESS THAN ( UNIX_TIMESTAMP('2010-01-01 00:00:00') ),
PARTITION p9 VALUES LESS THAN (MAXVALUE)
);
You can find it in:
https://dev.mysql.com/doc/refman/5.7/en/partitioning-range.html.
You can do this if you use DATE or DATETIME instead of TIMESTAMP as the data type.
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob DATETIME NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
Using the partition-selection hint is only supported in MySQL 5.6 and later.
See http://dev.mysql.com/doc/refman/5.6/en/partitioning-selection.html
Note that the page of the manual is only for MySQL 5.6. If you try to click on MySQL 5.5 documentation link, it redirects you back to 5.6.
The wrong is that, This field dob is not unique key!
you can use this command:
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL,
unique 'dob' (dob)
)
or the table is exist:
alter table t2 add UNIQUE(dob)
You can try it!

MySQL Sub-partitioning

I have the following MySQL table that is currently partitioned.
I want to add sub-partitioning on the 'message_read' key - this is to optimize searches through partitions for un-read messages.
CREATE TABLE `messages` (
`pid` int(50) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
`userid1` int(10) unsigned NOT NULL COMMENT 'Message Sender',
`userid2` int(10) unsigned NOT NULL COMMENT 'Message Receiver',
`message` text NOT NULL COMMENT 'Actual Message',
`message_read` tinyint(1) NOT NULL DEFAULT '0' COMMENT '0=Not Read & 1=Read',
`datetime` datetime NOT NULL COMMENT 'Date Time',
PRIMARY KEY (`pid`,`datetime`),
KEY `userid1` (`userid1`),
KEY `userid2` (`userid2`),
KEY `message_read` (`message_read`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
/*!50100 PARTITION BY RANGE (TO_DAYS(`datetime`))
(PARTITION p0 VALUES LESS THAN (735234) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (735265) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (735293) ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN (735324) ENGINE = InnoDB,
PARTITION p4 VALUES LESS THAN (735354) ENGINE = InnoDB,
PARTITION p5 VALUES LESS THAN (735385) ENGINE = InnoDB,
PARTITION p6 VALUES LESS THAN (735415) ENGINE = InnoDB) */
Current partition is on datetime stamp. How could I add into sub-partitioning on 'message_read'?
thankyou
You'd need to add message_read to the primary key.
Then (You can use hash or key for sub-partitioning):
CREATE TABLE `messages` (
`pid` INT(50) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
`userid1` INT(10) UNSIGNED NOT NULL COMMENT 'Message Sender',
`userid2` INT(10) UNSIGNED NOT NULL COMMENT 'Message Receiver',
`message` TEXT NOT NULL COMMENT 'Actual Message',
`message_read` TINYINT(1) NOT NULL DEFAULT '0' COMMENT '0=Not Read & 1=Read',
`datetime` DATETIME NOT NULL COMMENT 'Date Time',
PRIMARY KEY (`pid`,`datetime`, message_read),
KEY `userid1` (`userid1`),
KEY `userid2` (`userid2`),
KEY `message_read` (`message_read`)
) ENGINE=INNODB DEFAULT CHARSET=utf8
PARTITION BY RANGE (TO_DAYS(`datetime`))
SUBPARTITION BY HASH (message_read)
(PARTITION p0 VALUES LESS THAN (735234)(SUBPARTITION s0, SUBPARTITION s1),
PARTITION p1 VALUES LESS THAN (735265)(SUBPARTITION s2, SUBPARTITION s3),
PARTITION p2 VALUES LESS THAN (735293)(SUBPARTITION s4, SUBPARTITION s5),
PARTITION p3 VALUES LESS THAN (735324)(SUBPARTITION s6, SUBPARTITION s7),
PARTITION p4 VALUES LESS THAN (735354)(SUBPARTITION s8, SUBPARTITION s9),
PARTITION p5 VALUES LESS THAN (735385)(SUBPARTITION s10, SUBPARTITION s11),
PARTITION p6 VALUES LESS THAN (735415)(SUBPARTITION s12, SUBPARTITION s13)) ;
Hope this was what you were looking for.
Thanks,
y
In this case - Since each partition is devided into two parts,
How will mysql devide the record to the subpartitions?
after all mysql don't know how many records will be entered into each partition/subpartition