I've a 30M rows table and I want to partition it by dates.
mysql > SHOW CREATE TABLE `parameters`
CREATE TABLE `parameters` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`add_time` datetime DEFAULT NULL,
...(etc)
) ENGINE=MyISAM AUTO_INCREMENT=28929477 DEFAULT CHARSET=utf8
Table stores data for last 5 years and rows count increases dramatically. I want partition it by years(2009, 2010, 2011, 2012, 2013).
ALTER TABLE parameters DROP PRIMARY KEY, ADD INDEX(id);
ALTER TABLE parameters PARTITION BY RANGE (TO_DAYS(id)) (
PARTITION y2009 VALUES LESS THAN (TO_DAYS('2010-01-01')),
PARTITION y2010 VALUES LESS THAN (TO_DAYS('2011-01-01')),
PARTITION y2011 VALUES LESS THAN (TO_DAYS('2012-03-01')),
PARTITION y2012 VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION y2013 VALUES LESS THAN MAXVALUE
);
Everyting works on dev-server, but there is a problem on production-server.
The problem: almost all of the rows moved to the first partition(y2009). But data is uniformly distributed by years. Physically there is large y2009.myd file in DATA folder and others partitions have much less size.
Also I tried to reorganize first partition in order to exclude Null dates:
alter table raw
reorganize partition y2012 into (
PARTITION y0 VALUES LESS THAN (0),
PARTITION y2012 VALUES LESS THAN (TO_DAYS('2013-01-01')),
);
P.S.: production and dev servers have same version of MySQL 5.1.37
You need to use date column in RANGE not id for partition.
I have changed TO_DAYS(id) to TO_DAYS(add_time)
Try below:
ALTER TABLE parameters PARTITION BY RANGE (TO_DAYS(add_time)) (
PARTITION y0 VALUES LESS THAN (TO_DAYS('2009-01-01')),
PARTITION y2009 VALUES LESS THAN (TO_DAYS('2010-01-01')),
PARTITION y2010 VALUES LESS THAN (TO_DAYS('2011-01-01')),
PARTITION y2011 VALUES LESS THAN (TO_DAYS('2012-03-01')),
PARTITION y2012 VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION y2013 VALUES LESS THAN MAXVALUE
);
Related
As said, I'm doing and practicing a partition data in MySQL.
As I want to clear all the data, the data still there and cannot delete in one time.
This is the partition table that I created.
CREATE TABLE events
(
event_id INT unsigned NOT NULL AUTO_INCREMENT,
inserted_date DATETIME DEFAULT CURRENT_TIMESTAMP,
last_updated_date DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
data JSON NOT NULL,
KEY (event_id)
)
default charset utf8
PARTITION BY RANGE(to_days(inserted_date))
(
PARTITION p20210301 VALUES LESS THAN (to_days('2021-04-01')),
PARTITION p20210401 VALUES LESS THAN (to_days('2021-05-01')),
PARTITION p20210501 VALUES LESS THAN (to_days('2021-06-01')),
PARTITION p20210601 VALUES LESS THAN (to_days('2021-07-01')),
PARTITION p20210701 VALUES LESS THAN (to_days('2021-08-01')),
PARTITION p20210801 VALUES LESS THAN (to_days('2021-09-01')),
PARTITION p20210901 VALUES LESS THAN (to_days('2021-10-01')),
PARTITION p20211001 VALUES LESS THAN (to_days('2021-11-01')),
PARTITION p20211101 VALUES LESS THAN (to_days('2021-12-01')),
PARTITION p20211201 VALUES LESS THAN (to_days('2022-01-01')),
PARTITION p20220101 VALUES LESS THAN (to_days('2022-02-01')),
PARTITION p20220201 VALUES LESS THAN (to_days('2022-03-01')),
PARTITION p20220301 VALUES LESS THAN (to_days('2022-04-01')),
PARTITION p20220401 VALUES LESS THAN (to_days('2022-05-01')),
PARTITION p20220501 VALUES LESS THAN (to_days('2022-06-01')),
PARTITION future VALUES LESS THAN (MAXVALUE)
);
First question is everytime I select all data, different rows of data come out.
SELECT
*
FROM
plover_audit_log.events
Second I can't truncate the table. Everytime I truncate the table, the data still there.
truncate table plover_audit_log.events;
Third even I try to delete the data, it cannot delete completely. The data still there after delete.
delete from plover_audit_log.events;
I'm curious on is my MySQL software problem? Or partition table make this happen?
Because I have a backup table without partition is work well with Truncate, Delete and Select script.
data was running by one procedure, which found in processlist. So first need to kill that pid from processlist. like kill pid;
then execute truncate and check if data is still persist
Partitioned tables require a slightly different instruction to perform a TRUNCATE
To remove data from all partitions
ALTER plover_audit_log.events TRUNCATE PARTITION ALL;
To remove all data from a specific partition
ALTER plover_audit_log.events TRUNCATE PARTITION p20210301;
Reference https://dev.mysql.com/doc/refman/8.0/en/alter-table-partition-operations.html
I am exploring ways of partitioning a MySQL table by year and month. Can you please analyze my table creation below and see if this method of partitioning would end up putting data by month and year in these sub partitions? I'm using MySQL 5.5 and I can't use
SELECT * FROM points_log PARTITION (p0_p0sp0);
to validate if the partitioning is working. If there is a way to validate this in MySQL 5.5 please comment. I appreciate your feedback and criticisms on this table partitioning.
Here is my table creation:
CREATE TABLE `points_log` (
`id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`nick` char(25) NOT NULL,
`amount` decimal(7,4) NOT NULL,
`stream_online` tinyint(1) NOT NULL,
`modification_type` tinyint(3) unsigned NOT NULL,
`dt` datetime NOT NULL,
PRIMARY KEY (`id`,`dt`,`nick`),
KEY `nick_idx` (`nick`),
KEY `amount_idx` (`amount`),
KEY `modification_type_idx` (`modification_type`),
KEY `dt_idx` (`dt`),
KEY `stream_online_idx` (`stream_online`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=latin1
PARTITION BY RANGE( YEAR(dt) )
SUBPARTITION BY HASH( MONTH(dt) )
SUBPARTITIONS 12 (
PARTITION p0 VALUES LESS THAN (2014),
PARTITION p1 VALUES LESS THAN (2015),
PARTITION p2 VALUES LESS THAN (2016),
PARTITION p3 VALUES LESS THAN (2017),
PARTITION p4 VALUES LESS THAN (2018),
PARTITION p5 VALUES LESS THAN (2019),
PARTITION p6 VALUES LESS THAN (2020),
PARTITION p7 VALUES LESS THAN MAXVALUE
);
SUBPARTITIONs are probably useless. (That is, I have yet to find any advantage to their use. That especially applies to performance.)
Don't split the date; keep it as a single field.
Use BY RANGE(TO_DAYS(dt)) VALUES LESS THAN (TO_DAYS('2015-02-01'))
BY HASH is probably totally useless for performance.
WHERE dt BETWEEN .. AND .. cannot do partition pruning in the structure you have.
Do not use more than about 50 partitions (for performance reasons).
Do not create more than one 'future' partition; build them as needed. (This is a minor performance improvement.)
Do not use CHAR for variable length fields. Use VARCHAR.
I'm trying to alter an existing table to add year and week subpartitions, like so:
CREATE TABLE test_table(
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
dtime DATETIME NOT NULL);
ALTER TABLE test_table
PARTITION BY RANGE ( YEAR(dtime) )
SUBPARTITION BY RANGE( WEEK(dtime) ) (
PARTITION y0 VALUES LESS THAN (2013) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
),
PARTITION y1 VALUES LESS THAN (2014) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
),
PARTITION y2 VALUES LESS THAN (2015) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
),
PARTITION y3 VALUES LESS THAN (2016) (
SUBPARTITION w0 VALUES LESS THAN (2),
...
SUBPARTITION w52 VALUES LESS THAN (54)
)
);
However, this gives me the vague and unhelpful response of:
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'RANGE( WEEK(DTIME) ) (
PARTITION y0 VALUES LESS THAN (2013) (
SUBPARTITION ' at line 3
I've checked the docs: MySQL ALTER TABLE Partition operations and MySQL RANGE and LIST Partitions. However, neither of these describe how to alter a table to create subpartitions.
The second part of my question is for feedback on this partitioning scheme. The data that will go into this is sensor readings that are recorded every minute, and the most common query operation is for data in the last week. I think this should greatly speed up my queries, since a "WHERE dtime > date" is very common, without having to manually move data out of the table periodically into archive tables.
If you want to add a partition BY LIST to an already existing table, drop the primary key and create a composite primary key:
alter table test_table drop primary key, add primary key (id,<some other key>);
alter table orders partition by list(<some other key>) (
partition p0 values IN (1),
partition p1 values IN (2),
partition p2 values IN (3),
partition p3 values IN (4),
partition p4 values IN (5),
partition p5 values IN (6),
partition p6 values IN (7),
partition p7 values IN (8),
partition p8 values IN (9),
partition p9 values IN (10)
);
After further investigation, I have discovered several problems with this approach.
It is impossible to range partition on a DATETIME value (which dtime in the example is). http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-functions.html
The table I was partitioning had a primary key on an auto increment id column, and you cannot partition on an index if there is a different primary key.
ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the table's partitioning function
See also http://blog.mclaughlinsoftware.com/2011/05/09/mysqls-real-partition-key/
http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-partitioning-keys-unique-keys.html
WEEK() is not allowed as a partitioning function. http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-functions.html
From what I now know, if you have a UNIQUE AUTO_INCREMENT id as the primary key, it is impossible to partition on anything except that value.
My queries all use the dtime column in the WHERE conditions, so it seems that unless I can partition somehow on dtime still, there is no benefit to partitioning this table (from a performance perspective).
I am having issue to partition a table using partition by range on a datetime column.
the test search result is still on full partition scan.
I saw some posts on the net in regards to this issue, but not sure if there is any way to fix it or bypass the issue.
mysql server: Percona 5.5.24-55.
table:
id bigint(20) unsigned NOT NULL,
time datatime unsigned NOT NULL,
....
....
KEY id_time (id,time)
engine=InnoDB
partition statement:
alter table summary_201204
partition by range (day(time))
subpartition by key(id)
subpartitions 5 (
partition p0 values less than (6),
partition p1 values less than (11),
partition p2 values less than (16),
partition p3 values less than (21),
partition p4 values less than (26),
partition p5 values less than (MAXVALUE) );
check:
explain partitions select * from summary_201204 where time < '2012-07-21';
result: p0_p0sp0,p0_p0sp1,p0_p0sp2,p0_p0sp3,p0_p0sp4,p1_p1sp0,p1_p1sp1,p1_p1sp2,p1_p1sp3,p1_p1sp4,p2_p2sp0,p2_p2sp1,p2_p2sp2,p2_p2sp3,p2_p2sp4,p3_p3sp0,p3_p3sp1,p3_p3sp2,p3_p3sp3,p3_p3sp4,p4_p4sp0,p4_p4sp1,p4_p4sp2,p4_p4sp3,p4_p4sp4,p5_p5sp0,p5_p5sp1,p5_p5sp2,p5_p5sp3,p5_p5sp4.
I think here is the answer: Visit enter link description here
So, the documentation within the mysql official site is not clear enough about the data types required for partition. In this case, if the table data type is datetime, then we should use to_seconds, whilst if the data type is DATE then we can use YEA
I want to use mysql partition tables to partition a table into YEAR and the WEEK number. I know exactly how to do this with mysql merge tables but partition tables are different. Can someone please help with the following table schema?
CREATE TABLE `tableName` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`dateandtime` datetime NOT NULL,
`othervalue` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
Also does it have to be in a certain engine?
And if I store the dateandtime as a int(10) timestamp how would I do it?
CREATE TABLE `tableName` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`dateandtime` int(10) NOT NULL,
`othervalue` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
MySQL 5.1 cannot do partition by date, so you must you workaround... Usually you partition by function TO_DAYS(dateandtime), for example like this:
CREATE TABLE tbl (
... ) ENGINE=InnoDB
PARTITION BY RANGE (to_days(dateandtime)) (
PARTITION pNULL VALUES LESS THAN (0) ENGINE = InnoDB,
PARTITION p20111218 VALUES LESS THAN (TO_DAYS('2011-12-18')) ENGINE = InnoDB,
PARTITION p20111225 VALUES LESS THAN (TO_DAYS('2011-12-25')) ENGINE = InnoDB,
PARTITION pNew VALUES LESS THAN MAXVALUE ENGINE = InnoDB
)
I defined here 4 partitions - the first is just for sake of completeness, so that insert of date in future won't fail. (You can't INSERT a value for which a partition does not exist.) The first partition is for performance - NULL values will be petentionally stored there. The middle 2 partitions are actually being used, each keeping one week of data.
You can drop old partition (this is very fast compared to just DELETEing old rows) using ALTER TABLE tbl DROP PARTITION xyz. You can add new partitions by splitting the last partition:
ALTER TABLE tbl REORGANIZE PARTITION pNew INTO (
PARTITION p20120115 VALUES LESS THAN (TO_DAYS('2012-01-16')),
...
PARTITION pNew VALUES LESS THAN (MAXVALUE)
);