MySQL table partitioning on timestamp - mysql

I have partitioned a table (because of an out of memory error - table got too big). I have partitioned it on a timestamp column as shown below:
CREATE TABLE test (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( unix_timestamp(dob) ) (
PARTITION p2012 VALUES LESS THAN (unix_timestamp('2013-01-01 00:00:00')),
PARTITION p2013 VALUES LESS THAN (unix_timestamp('2014-01-01 00:00:00')),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
I was hoping that the process of partitioning would also help in speeding up a couple of my queries whihc take a few hours to run; however, this type of partitioning doesn't seem to kick in and all partitions are still being used and scanned through for the queries. I have tried, and failed, with a couple more approaches:
1) Tried to use different range for the partitioning
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
However, that results in an error: Error Code: 1486. Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
2) Gave up on changing partitioning to be recognized by the query optimizer, and as suggested in MySQL's Doc - 18.5 Partition Selection tried specifying which partitions to use in the select statement instead:
select * from t2 partition (p2012)
But, that returns a syntax error Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(p2012) LIMIT 0, 1000' at line 1
Does anybody have any suggestions what else I could try to utilize table partitioning to optimize the queries?

You can use UNIX_TIMESTAMP() function. Example from MySQL docs:
CREATE TABLE quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-07-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-10-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-01-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-04-01 00:00:00') ),
PARTITION p6 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-07-01 00:00:00') ),
PARTITION p7 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-10-01 00:00:00') ),
PARTITION p8 VALUES LESS THAN ( UNIX_TIMESTAMP('2010-01-01 00:00:00') ),
PARTITION p9 VALUES LESS THAN (MAXVALUE)
);
You can find it in:
https://dev.mysql.com/doc/refman/5.7/en/partitioning-range.html.

You can do this if you use DATE or DATETIME instead of TIMESTAMP as the data type.
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob DATETIME NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
Using the partition-selection hint is only supported in MySQL 5.6 and later.
See http://dev.mysql.com/doc/refman/5.6/en/partitioning-selection.html
Note that the page of the manual is only for MySQL 5.6. If you try to click on MySQL 5.5 documentation link, it redirects you back to 5.6.

The wrong is that, This field dob is not unique key!
you can use this command:
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL,
unique 'dob' (dob)
)
or the table is exist:
alter table t2 add UNIQUE(dob)
You can try it!

Related

How to create partition mysql 8

I am trying to create a table with partition with following way
create table transaction_demo
(
trx_id bigint not null primary key,
created_datetime datetime
) partition by range (YEAR(created_datetime)) (
partition p1 values less than (2019),
partition p2 values less than (2020),
partition p3 values less than (2021)
);
But it is not working. I am getting following error
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
How to fix this?
Then define primary key for each column which will be part of the partition.
CREATE TABLE transaction_demo (
trx_id int NOT NULL,
created_datetime datetime NOT NULL,
other_column_1 int NULL,
other_column_2 varchar(222) NULL,
PRIMARY KEY(trx_id,created_datetime)
)
PARTITION BY RANGE( YEAR(created_datetime) )(
PARTITION p1 VALUES LESS THAN (2019),
PARTITION p2 VALUES LESS THAN (2020),
PARTITION p3 VALUES LESS THAN (2021)
);

Mysql - how to keep unique constraint while partitioning by RANGE (timestamp)?

I have one table, want to partition by RANGE (created_at timestamp), so can delete old data easily (by drop partition).
CREATE TABLE `orders` (
`order_id` NVARCHAR(64) NOT NULL,
`amount` INTEGER NOT NULL,
`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
`modified_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
KEY `order_id` (`order_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
ALTER TABLE dropship.orders
PARTITION BY RANGE (UNIX_TIMESTAMP(created_at)) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-03-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-05-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-06-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-07-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN (MAXVALUE)
);
This table only has two usages: get by order_id, or update by order_id.
select * from orders where order_id = '123';
update orders set amount = 10 where order_id = '123';
Due to the limitation of Mysql partitioning, I cannot add an unique key for order_id since will use created_at field for partitioning.
All columns used in the table's partitioning expression must be part of every unique key that the table may have, including any primary key.
Question:
Any way to make order_id unique in this table please?
I have thought about partitioning by order_id, but it's hard to delete old data in that way.
Any suggestion is welcome. (For example may be you have better design for this table).
BEGIN;
SELECT 1 FROM orders WHERE order_id = 234 FOR UPDATE;
if row exists, you have a dup error.
INSERT INTO orders ... order_id = 234;
COMMIT;
But, as Raymond points out, you may as well drop PARTITIONing and make the column the PRIMARY KEY. This would make all the stated operations slightly faster.

MySQL Partitioning Error

I'm running MySQL 5.6 and have the following table:
CREATE TABLE `rawdata` (
`RawDataSeq` int(10) unsigned NOT NULL AUTO_INCREMENT,
`SpeciesSeq` int(10) unsigned DEFAULT NULL,
`DataSourceSeq` int(10) unsigned DEFAULT NULL,
`MeasurementSeq` int(10) unsigned DEFAULT NULL,
`RecordingDateTime` datetime DEFAULT NULL,
`RawSensorData` varchar(100) DEFAULT NULL,
`RawDataNumeric` decimal(36,18) DEFAULT NULL,
`RawDataString` varchar(100) DEFAULT NULL,
`RawDataDate` datetime DEFAULT NULL,
`RawDataTime` varchar(45) DEFAULT NULL,
`RawDataDateTime` datetime DEFAULT NULL,
`ActiveRecord` int(10) unsigned DEFAULT NULL,
`LastUpdateDate` datetime DEFAULT NULL,
`UserId` varchar(6) DEFAULT NULL,
`BadData` varchar(200) DEFAULT NULL,
PRIMARY KEY (`RawDataSeq`)
) ENGINE=MyISAM AUTO_INCREMENT=5676258 DEFAULT CHARSET=latin1
I'm trying to partition the table by 'RecordingDateTime' using the following command:
ALTER TABLE rawdata PARTITION BY RANGE (RecordingDateTime) (
PARTITION p0 VALUES LESS THAN (2013),
PARTITION p1 VALUES LESS THAN (2014),
PARTITION p2 VALUES LESS THAN (2015),
PARTITION p3 VALUES LESS THAN (UNIX_TIMESTAMP('2015-06-01 00:00:00')),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
I'm receiving the following error message:
Query: ALTER TABLE rawdata PARTITION BY RANGE (RecordingDateTime) ( PARTITION p0 VALUES LESS THAN (2013), PARTITION p1 VALUES LESS THAN...
Error Code: 1659
Field 'RecordingDateTime' is of a not allowed type for this type of partitioning
Any help is greatly appreciated.
Change to
PRIMARY KEY (`RawDataSeq`, RecordingDateTime)
ALTER TABLE rawdata PARTITION BY RANGE (TO_DAYS(RecordingDateTime)) (
PARTITION p_old VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION p2013 VALUES LESS THAN (TO_DAYS('2014-01-01')),
PARTITION p2014 VALUES LESS THAN (TO_DAYS('2015-01-01')),
PARTITION p2015 VALUES LESS THAN (TO_DAYS('2016-01-01')),
PARTITION future VALUES LESS THAN MAXVALUE
);
The code you need each December is found here.
I suspect there is a problem with the datetime value and the integer values you are using. One method is:
ALTER TABLE rawdata PARTITION BY RANGE (year(RecordingDateTime)) (
PARTITION p0 VALUES LESS THAN (2013),
PARTITION p1 VALUES LESS THAN (2014),
PARTITION p2 VALUES LESS THAN (2015),
PARTITION p3 VALUES LESS THAN MAXVALUE
);
If you don't store the RecordingDateTime as a timestamp, you can also try:
ALTER TABLE rawdata PARTITION BY RANGE (year(RecordingDateTime)*100 + month(RecordDateTime) (
PARTITION p0 VALUES LESS THAN (201300),
PARTITION p1 VALUES LESS THAN (201400),
PARTITION p2 VALUES LESS THAN (201500),
PARTITION p3 VALUES LESS THAN (201506),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
Otherwise, store the value as a timestamp and use timestamp ranges.

How do I create a partition for every day of the current year in mysql table

Actually I want to store data last three month from current date rest of record deleted. data having millions of record
you can use given query.
CREATE TABLE TABLE_NAME1 (
id BIGINT(20) NOT NULL AUTO_INCREMENT,
Ctime DATETIME DEFAULT NULL,
KEY id (id)
) ENGINE=INNODB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY RANGE( TO_DAYS(Ctime) ) (
PARTITION p1 VALUES LESS THAN (TO_DAYS('2011-04-02')),
PARTITION p2 VALUES LESS THAN (TO_DAYS('2011-04-03')),
PARTITION p3 VALUES LESS THAN (TO_DAYS('2011-04-04')),
PARTITION p4 VALUES LESS THAN (TO_DAYS('2011-04-05')),
PARTITION p5 VALUES LESS THAN (TO_DAYS('2011-04-06')),
PARTITION p6 VALUES LESS THAN (TO_DAYS('2011-04-07'))
);

partitions and sub partitions

The following CREATE TABLE statement to partition a table works as expected, without error.
CREATE TABLE `ox_data_archive_20120108` (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY LIST(to_days(datetime )) (
PARTITION `1Jan10` VALUES IN (to_days('2010-01-01') ),
PARTITION `2Jan10` VALUES IN (to_days('2010-01-02') ),
PARTITION `3Jan10` VALUES IN (to_days('2010-01-03') )
);
What I need to do is to create subpartitions based on date + zoneid. I tried the following:
CREATE TABLE mypart (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY LIST(to_days(datetime ))
SUBPARTITION BY KEY(zoneid) (
PARTITION `1Jan10` VALUES IN (to_days('2010-01-01') )
(Subpartition s1, Subpartition s2),
PARTITION `2Jan10` VALUES IN (to_days('2010-01-02') )
(Subpartition s3, Subpartition s4),
PARTITION `3Jan10` VALUES IN (to_days('2010-01-03') )
(Subpartition s5, Subpartition s6)
);
Inserting into this table:
INSERT INTO mypart VALUES (1, 2, 3, '2012-01-31 04:10:03');
results in the following error:
ERROR 1526 (HY000): Table has no partition for value 734898
My query expects to use the zoneid subpartition based on dates. Is it possible?
Contrary to your assertion that the first table works without error, inserting the sample data into it:
INSERT INTO `ox_data_archive_20120108` VALUES (1, 2, 3, '2012-01-31 04:10:03');
results in the same error as for the second table. The value given in the error (734898) happens to be the value for to_days('2012-01-31'). You get this error because you only have partitions for January 1st through 3rd, 2010. Both the day-of-month and the year for the sample data are outside the defined partitions. Instead of TO_DAYS (which returns the number of days from year 0 to the given date), you probably want DAYOFMONTH. Since each partition is contiguous, a RANGE partition seems more appropriate than a LIST.
Off topic, you only need to specify separate subpartition definitions when you want to set options for the subpartitions. Since you're not doing that, a SUBPARTITIONS 2 clause will do the same thing as your statement, but is simpler.
CREATE TABLE mypart (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY RANGE(DAYOFMONTH(`datetime`))
SUBPARTITION BY KEY(zoneid)
SUBPARTITIONS 2 (
PARTITION `01` VALUES LESS THAN 2, -- Note: 0 is valid day-of-month
PARTITION `02` VALUES LESS THAN 3,
PARTITION `03` VALUES LESS THAN 4,
...
);