I have a table which contains 50GB data now I am trying to partition on the datetime column below are the details.
CREATE TABLE EDR2 ( id varchar(255),appKey varchar(255),clickedTime datetime ,ettId bigint(20),primary key(id),Key `ettId` (ettId),Key `clickedTime_index` (`clickedTime`) )
PARTITION BY RANGE (day(clickedTime))
( PARTITION p01 VALUES LESS THAN (2) ,
PARTITION p02 VALUES LESS THAN (3) ,
PARTITION p03 VALUES LESS THAN (4) ,
PARTITION p04 VALUES LESS THAN (5) ,
PARTITION p05 VALUES LESS THAN (6) ,
PARTITION p06 VALUES LESS THAN (7) ,
PARTITION p07 VALUES LESS THAN (8) ,
PARTITION p08 VALUES LESS THAN (9) ,
PARTITION p09 VALUES LESS THAN (10) ,
PARTITION p10 VALUES LESS THAN (11) ,
PARTITION p11 VALUES LESS THAN (12) ,
PARTITION p12 VALUES LESS THAN (13) ,
PARTITION p13 VALUES LESS THAN (14) ,
PARTITION p14 VALUES LESS THAN (15) ,
PARTITION p15 VALUES LESS THAN (16) ,
PARTITION p16 VALUES LESS THAN (17) ,
PARTITION p17 VALUES LESS THAN (18) ,
PARTITION p18 VALUES LESS THAN (19) ,
PARTITION p19 VALUES LESS THAN (20) ,
PARTITION p20 VALUES LESS THAN (21) ,
PARTITION p21 VALUES LESS THAN (22) ,
PARTITION p22 VALUES LESS THAN (23) ,
PARTITION p23 VALUES LESS THAN (24) ,
PARTITION p24 VALUES LESS THAN (25) ,
PARTITION p25 VALUES LESS THAN (26) ,
PARTITION p26 VALUES LESS THAN (27) ,
PARTITION p27 VALUES LESS THAN (28) ,
PARTITION p28 VALUES LESS THAN (29) ,
PARTITION p29 VALUES LESS THAN (30) ,
PARTITION p30 VALUES LESS THAN (31) ,
PARTITION p31 VALUES LESS THAN MAXVALUE);*
and I am getting the error:-
***ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the table's partitioning function***
I can't create a composite primary key in (id,clickedTime) and can not remove the primary key, because it will allow the duplicate records,
I don't want to change anything in my code.
so please suggest.
Any UNIQUE key on a partitioned table must include the partition key because otherwise the UNIQUEness constraint would be painfully costly. Each INSERT would have to check all partitions to verify that the constraint is not violated.
Considering that PARTITIONing is normally useful only on very large tables, this overhead would be deadly.
You are partitioning on day of month - Why? What do you hope to gain? Checking for a "range" on clickedTime will not do "partition pruning" (I think). See EXPLAIN PARTITIONS SELECT ...
What is the ID like, for it to be VARCHAR(255)? Big PRIMARY KEYs are generally not practical.
Is this table MyISAM or InnoDB?
I recommend getting rid of the partitioning.
What are some of the SELECTs? I see no compound INDEXes currently, usually it is prudent to have a compound index ending with a DATETIME.
Related
I've come across a couple of questions like my question:
MySQL table partitioning on timestamp
MySQL Database Error: Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
And some other. They all explained the same problem but they weren't my issue.
My problem is that my column is datetime NOT NULL and I have made it a primary key by table id:
PRIMARY KEY (id, start_time)
Now I used the below command to partition my table:
alter table MY_TABLE partition by range (UNIX_TIMESTAMP(start_time)) (
partition p2012 values less than (UNIX_TIMESTAMP('2012-01-01 00:00:00')),
partition p2013 values less than (UNIX_TIMESTAMP('2013-01-01 00:00:00')),
partition p2014 values less than (UNIX_TIMESTAMP('2014-01-01 00:00:00')),
partition p2015 values less than (UNIX_TIMESTAMP('2015-01-01 00:00:00')),
partition p2016 values less than (UNIX_TIMESTAMP('2016-01-01 00:00:00')),
partition p2017 values less than (UNIX_TIMESTAMP('2017-01-01 00:00:00')),
partition pfuture values less than (MAXVALUE));
Everything seems right, and When I describe MY_TABLE it gives information of:
start_time | datetime | NO | PRI | 0000-00-00 00:00:00
When run the command above for partitioning I get the error of:
ERROR 1486 (HY000): Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
datetime is not timezone dependent like timestamp, and it is not constant. What could be the reason for such an error?
22.2.1 RANGE Partitioning
...
Partition the table by RANGE COLUMNS, using a DATE or DATETIME column
as the partitioning column.
...
Try:
ALTER TABLE `MY_TABLE`
PARTITION BY RANGE COLUMNS (`start_time`)
(
PARTITION `p2012` VALUES LESS THAN ('2012-01-01 00:00:00'),
PARTITION `p2013` VALUES LESS THAN ('2013-01-01 00:00:00'),
PARTITION `p2014` VALUES LESS THAN ('2014-01-01 00:00:00'),
PARTITION `p2015` VALUES LESS THAN ('2015-01-01 00:00:00'),
PARTITION `p2016` VALUES LESS THAN ('2016-01-01 00:00:00'),
PARTITION `p2017` VALUES LESS THAN ('2017-01-01 00:00:00'),
PARTITION `pfuture` VALUES LESS THAN (MAXVALUE)
);
See db-fiddle.
I am exploring ways of partitioning a MySQL table by year and month. Can you please analyze my table creation below and see if this method of partitioning would end up putting data by month and year in these sub partitions? I'm using MySQL 5.5 and I can't use
SELECT * FROM points_log PARTITION (p0_p0sp0);
to validate if the partitioning is working. If there is a way to validate this in MySQL 5.5 please comment. I appreciate your feedback and criticisms on this table partitioning.
Here is my table creation:
CREATE TABLE `points_log` (
`id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`nick` char(25) NOT NULL,
`amount` decimal(7,4) NOT NULL,
`stream_online` tinyint(1) NOT NULL,
`modification_type` tinyint(3) unsigned NOT NULL,
`dt` datetime NOT NULL,
PRIMARY KEY (`id`,`dt`,`nick`),
KEY `nick_idx` (`nick`),
KEY `amount_idx` (`amount`),
KEY `modification_type_idx` (`modification_type`),
KEY `dt_idx` (`dt`),
KEY `stream_online_idx` (`stream_online`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=latin1
PARTITION BY RANGE( YEAR(dt) )
SUBPARTITION BY HASH( MONTH(dt) )
SUBPARTITIONS 12 (
PARTITION p0 VALUES LESS THAN (2014),
PARTITION p1 VALUES LESS THAN (2015),
PARTITION p2 VALUES LESS THAN (2016),
PARTITION p3 VALUES LESS THAN (2017),
PARTITION p4 VALUES LESS THAN (2018),
PARTITION p5 VALUES LESS THAN (2019),
PARTITION p6 VALUES LESS THAN (2020),
PARTITION p7 VALUES LESS THAN MAXVALUE
);
SUBPARTITIONs are probably useless. (That is, I have yet to find any advantage to their use. That especially applies to performance.)
Don't split the date; keep it as a single field.
Use BY RANGE(TO_DAYS(dt)) VALUES LESS THAN (TO_DAYS('2015-02-01'))
BY HASH is probably totally useless for performance.
WHERE dt BETWEEN .. AND .. cannot do partition pruning in the structure you have.
Do not use more than about 50 partitions (for performance reasons).
Do not create more than one 'future' partition; build them as needed. (This is a minor performance improvement.)
Do not use CHAR for variable length fields. Use VARCHAR.
I've a 30M rows table and I want to partition it by dates.
mysql > SHOW CREATE TABLE `parameters`
CREATE TABLE `parameters` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`add_time` datetime DEFAULT NULL,
...(etc)
) ENGINE=MyISAM AUTO_INCREMENT=28929477 DEFAULT CHARSET=utf8
Table stores data for last 5 years and rows count increases dramatically. I want partition it by years(2009, 2010, 2011, 2012, 2013).
ALTER TABLE parameters DROP PRIMARY KEY, ADD INDEX(id);
ALTER TABLE parameters PARTITION BY RANGE (TO_DAYS(id)) (
PARTITION y2009 VALUES LESS THAN (TO_DAYS('2010-01-01')),
PARTITION y2010 VALUES LESS THAN (TO_DAYS('2011-01-01')),
PARTITION y2011 VALUES LESS THAN (TO_DAYS('2012-03-01')),
PARTITION y2012 VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION y2013 VALUES LESS THAN MAXVALUE
);
Everyting works on dev-server, but there is a problem on production-server.
The problem: almost all of the rows moved to the first partition(y2009). But data is uniformly distributed by years. Physically there is large y2009.myd file in DATA folder and others partitions have much less size.
Also I tried to reorganize first partition in order to exclude Null dates:
alter table raw
reorganize partition y2012 into (
PARTITION y0 VALUES LESS THAN (0),
PARTITION y2012 VALUES LESS THAN (TO_DAYS('2013-01-01')),
);
P.S.: production and dev servers have same version of MySQL 5.1.37
You need to use date column in RANGE not id for partition.
I have changed TO_DAYS(id) to TO_DAYS(add_time)
Try below:
ALTER TABLE parameters PARTITION BY RANGE (TO_DAYS(add_time)) (
PARTITION y0 VALUES LESS THAN (TO_DAYS('2009-01-01')),
PARTITION y2009 VALUES LESS THAN (TO_DAYS('2010-01-01')),
PARTITION y2010 VALUES LESS THAN (TO_DAYS('2011-01-01')),
PARTITION y2011 VALUES LESS THAN (TO_DAYS('2012-03-01')),
PARTITION y2012 VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION y2013 VALUES LESS THAN MAXVALUE
);
I'm trying to alter an existing table to add year and week subpartitions, like so:
CREATE TABLE test_table(
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
dtime DATETIME NOT NULL);
ALTER TABLE test_table
PARTITION BY RANGE ( YEAR(dtime) )
SUBPARTITION BY RANGE( WEEK(dtime) ) (
PARTITION y0 VALUES LESS THAN (2013) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
),
PARTITION y1 VALUES LESS THAN (2014) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
),
PARTITION y2 VALUES LESS THAN (2015) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
),
PARTITION y3 VALUES LESS THAN (2016) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
)
);
However, this gives me the vague and unhelpful response of:
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'RANGE( WEEK(DTIME) ) (
PARTITION y0 VALUES LESS THAN (2013) (
SUBPARTITION ' at line 3
I've checked the docs: MySQL ALTER TABLE Partition operations and MySQL RANGE and LIST Partitions. However, neither of these describe how to alter a table to create subpartitions.
The second part of my question is for feedback on this partitioning scheme. The data that will go into this is sensor readings that are recorded every minute, and the most common query operation is for data in the last week. I think this should greatly speed up my queries, since a "WHERE dtime > date" is very common, without having to manually move data out of the table periodically into archive tables.
If you want to add a partition BY LIST to an already existing table, drop the primary key and create a composite primary key:
alter table test_table drop primary key, add primary key (id,<some other key>);
alter table orders partition by list(<some other key>) (
partition p0 values IN (1),
partition p1 values IN (2),
partition p2 values IN (3),
partition p3 values IN (4),
partition p4 values IN (5),
partition p5 values IN (6),
partition p6 values IN (7),
partition p7 values IN (8),
partition p8 values IN (9),
partition p9 values IN (10)
);
After further investigation, I have discovered several problems with this approach.
It is impossible to range partition on a DATETIME value (which dtime in the example is). http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-functions.html
The table I was partitioning had a primary key on an auto increment id column, and you cannot partition on an index if there is a different primary key.
ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the table's partitioning function
See also http://blog.mclaughlinsoftware.com/2011/05/09/mysqls-real-partition-key/
http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-partitioning-keys-unique-keys.html
WEEK() is not allowed as a partitioning function. http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-functions.html
From what I now know, if you have a UNIQUE AUTO_INCREMENT id as the primary key, it is impossible to partition on anything except that value.
My queries all use the dtime column in the WHERE conditions, so it seems that unless I can partition somehow on dtime still, there is no benefit to partitioning this table (from a performance perspective).
I am having issue to partition a table using partition by range on a datetime column.
the test search result is still on full partition scan.
I saw some posts on the net in regards to this issue, but not sure if there is any way to fix it or bypass the issue.
mysql server: Percona 5.5.24-55.
table:
id bigint(20) unsigned NOT NULL,
time datatime unsigned NOT NULL,
....
....
KEY id_time (id,time)
engine=InnoDB
partition statement:
alter table summary_201204
partition by range (day(time))
subpartition by key(id)
subpartitions 5 (
partition p0 values less than (6),
partition p1 values less than (11),
partition p2 values less than (16),
partition p3 values less than (21),
partition p4 values less than (26),
partition p5 values less than (MAXVALUE) );
check:
explain partitions select * from summary_201204 where time < '2012-07-21';
result: p0_p0sp0,p0_p0sp1,p0_p0sp2,p0_p0sp3,p0_p0sp4,p1_p1sp0,p1_p1sp1,p1_p1sp2,p1_p1sp3,p1_p1sp4,p2_p2sp0,p2_p2sp1,p2_p2sp2,p2_p2sp3,p2_p2sp4,p3_p3sp0,p3_p3sp1,p3_p3sp2,p3_p3sp3,p3_p3sp4,p4_p4sp0,p4_p4sp1,p4_p4sp2,p4_p4sp3,p4_p4sp4,p5_p5sp0,p5_p5sp1,p5_p5sp2,p5_p5sp3,p5_p5sp4.
I think here is the answer: Visit enter link description here
So, the documentation within the mysql official site is not clear enough about the data types required for partition. In this case, if the table data type is datetime, then we should use to_seconds, whilst if the data type is DATE then we can use YEA