I want to use mysql partition tables to partition a table into YEAR and the WEEK number. I know exactly how to do this with mysql merge tables but partition tables are different. Can someone please help with the following table schema?
CREATE TABLE `tableName` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`dateandtime` datetime NOT NULL,
`othervalue` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
Also does it have to be in a certain engine?
And if I store the dateandtime as a int(10) timestamp how would I do it?
CREATE TABLE `tableName` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`dateandtime` int(10) NOT NULL,
`othervalue` int(10) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
MySQL 5.1 cannot do partition by date, so you must you workaround... Usually you partition by function TO_DAYS(dateandtime), for example like this:
CREATE TABLE tbl (
... ) ENGINE=InnoDB
PARTITION BY RANGE (to_days(dateandtime)) (
PARTITION pNULL VALUES LESS THAN (0) ENGINE = InnoDB,
PARTITION p20111218 VALUES LESS THAN (TO_DAYS('2011-12-18')) ENGINE = InnoDB,
PARTITION p20111225 VALUES LESS THAN (TO_DAYS('2011-12-25')) ENGINE = InnoDB,
PARTITION pNew VALUES LESS THAN MAXVALUE ENGINE = InnoDB
)
I defined here 4 partitions - the first is just for sake of completeness, so that insert of date in future won't fail. (You can't INSERT a value for which a partition does not exist.) The first partition is for performance - NULL values will be petentionally stored there. The middle 2 partitions are actually being used, each keeping one week of data.
You can drop old partition (this is very fast compared to just DELETEing old rows) using ALTER TABLE tbl DROP PARTITION xyz. You can add new partitions by splitting the last partition:
ALTER TABLE tbl REORGANIZE PARTITION pNew INTO (
PARTITION p20120115 VALUES LESS THAN (TO_DAYS('2012-01-16')),
...
PARTITION pNew VALUES LESS THAN (MAXVALUE)
);
Related
This is my table schema.
CREATE TABLE users (
`id` int(11) NOT NULL AUTO_INCREMENT,
`created_at` datetime DEFAULT NULL,
`account_id` tinyint(4) NOT NULL,
) ENGINE=InnoDB AUTO_INCREMENT=25600033 DEFAULT CHARSET=utf8
PARTITION BY LIST (account_id)
(PARTITION p0 VALUES IN (1) ENGINE = InnoDB,
PARTITION p1 VALUES IN (2) ENGINE = InnoDB,
PARTITION p2 VALUES IN (3) ENGINE = InnoDB)
The query is
select * from users where account_id in (1,2);
Does sql server will check in partion 1 & 2 parallely or one by one??
Yes, one by one.
There is no parallelism in a single connection in MySQL. Not for UNION, not for PARTITIONs. Not (so far) in version 8.0.
There is probably no performance to be gained by PARTITION BY LIST. Further comments: http://mysql.rjweb.org/doc.php/partitionmaint
I am exploring ways of partitioning a MySQL table by year and month. Can you please analyze my table creation below and see if this method of partitioning would end up putting data by month and year in these sub partitions? I'm using MySQL 5.5 and I can't use
SELECT * FROM points_log PARTITION (p0_p0sp0);
to validate if the partitioning is working. If there is a way to validate this in MySQL 5.5 please comment. I appreciate your feedback and criticisms on this table partitioning.
Here is my table creation:
CREATE TABLE `points_log` (
`id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`nick` char(25) NOT NULL,
`amount` decimal(7,4) NOT NULL,
`stream_online` tinyint(1) NOT NULL,
`modification_type` tinyint(3) unsigned NOT NULL,
`dt` datetime NOT NULL,
PRIMARY KEY (`id`,`dt`,`nick`),
KEY `nick_idx` (`nick`),
KEY `amount_idx` (`amount`),
KEY `modification_type_idx` (`modification_type`),
KEY `dt_idx` (`dt`),
KEY `stream_online_idx` (`stream_online`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=latin1
PARTITION BY RANGE( YEAR(dt) )
SUBPARTITION BY HASH( MONTH(dt) )
SUBPARTITIONS 12 (
PARTITION p0 VALUES LESS THAN (2014),
PARTITION p1 VALUES LESS THAN (2015),
PARTITION p2 VALUES LESS THAN (2016),
PARTITION p3 VALUES LESS THAN (2017),
PARTITION p4 VALUES LESS THAN (2018),
PARTITION p5 VALUES LESS THAN (2019),
PARTITION p6 VALUES LESS THAN (2020),
PARTITION p7 VALUES LESS THAN MAXVALUE
);
SUBPARTITIONs are probably useless. (That is, I have yet to find any advantage to their use. That especially applies to performance.)
Don't split the date; keep it as a single field.
Use BY RANGE(TO_DAYS(dt)) VALUES LESS THAN (TO_DAYS('2015-02-01'))
BY HASH is probably totally useless for performance.
WHERE dt BETWEEN .. AND .. cannot do partition pruning in the structure you have.
Do not use more than about 50 partitions (for performance reasons).
Do not create more than one 'future' partition; build them as needed. (This is a minor performance improvement.)
Do not use CHAR for variable length fields. Use VARCHAR.
I have created a table using mysql partition using range and have inserted millions of data.
CREATE TABLE `PART_SAMPLE ` (
`TRANSACTION_ID` bigint(25) NOT NULL AUTO_INCREMENT,
`TASK_ID` int(11) DEFAULT NULL,
`STATUS_CODE` int(10) DEFAULT NULL,
`FIELD10` int(5) DEFAULT NULL,
KEY `TXN_ID` (`TRANSACTION_ID`),
KEY `TASK_IDX` (`TASK_ID`),
KEY `id_idx_task_status` (`TASK_ID`,`STATUS_CODE`),
KEY `IDX_STATUS` (`STATUS_CODE`),
KEY `Fld_idx` (`FIELD10`)
) ENGINE=MyISAM AUTO_INCREMENT=12249932 DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (FIELD10)
(PARTITION p0 VALUES LESS THAN (0) ENGINE = MyISAM,
PARTITION p1 VALUES LESS THAN (1) ENGINE = MyISAM,
PARTITION p2 VALUES LESS THAN (2) ENGINE = MyISAM,
........
PARTITION p9 VALUES LESS THAN (9) ENGINE = MyISAM,
PARTITION p10 VALUES LESS THAN MAXVALUE ENGINE = MyISAM) */
Each Field10(0-10) value is having 3 million data each.
But when am executing a select query as this
select TASK_ID,STATUS_CODE,count(*) from PART_SAMPLE where FIELD10=X group by TASK_ID,STATUS_CODE;
x can be any value in the partition
for x value 0,2,5,8 it is taking only 10 seconds to retrive result but for rest it is taking abount 50s to rerive the result. As per my understating since data is same for all Fields almost same time has to be taken for any Field10 value. Why this time difference is coming
I update MySQL versition from 5.0 to 5.5. and I am new for studying mysql partition. firstly, I type:
SHOW VARIABLES LIKE '%partition%'
Variable_name Value
have_partitioning YES
Make sure that the new version support partition. I tried to partition my table by every 10 minutes, then INSERT, UPDATE, QUERY huge data into this table for a test.
First, I need create a new table, I type my code:
CREATE TABLE test (
`id` int unsigned NOT NULL auto_increment,
`words` varchar(100) collate utf8_unicode_ci NOT NULL,
`date` varchar(10) collate utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `index` (`words`)
)
ENGINE=MyISAM
DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci
AUTO_INCREMENT=0
PARTITION BY RANGE (MINUTE(`date`))
(
PARTITION p0 VALUES LESS THAN (1322644000),
PARTITION p1 VALUES LESS THAN (1322644600) ,
PARTITION p2 VALUES LESS THAN (1322641200) ,
PARTITION p3 VALUES LESS THAN (1322641800) ,
PARTITION p4 VALUES LESS THAN MAXVALUE
);
It return alert: #1564 - This partition function is not allowed, so what is this problem? thanks.
UPDATE
Modify date into int NOT NULL, and PARTITION BY RANGE MINUTE(date) into PARTITION BY RANGE COLUMNS(date)
CREATE TABLE test (
`id` int unsigned NOT NULL auto_increment,
`words` varchar(100) collate utf8_unicode_ci NOT NULL,
`date` int NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `index` (`words`)
)
ENGINE=MyISAM
DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci
AUTO_INCREMENT=0
PARTITION BY RANGE COLUMNS(`date`)
(
PARTITION p0 VALUES LESS THAN (1322644000),
PARTITION p1 VALUES LESS THAN (1322644600) ,
PARTITION p2 VALUES LESS THAN (1322641200) ,
PARTITION p3 VALUES LESS THAN (1322641800) ,
PARTITION p4 VALUES LESS THAN MAXVALUE
);
Then caused new error: #1214 - The used table type doesn't support FULLTEXT indexes
I am so sorry, mysql not support fulltext and partition at the same time.
See partitioning limitations
FULLTEXT indexes. Partitioned tables do not support FULLTEXT indexes or searches. This includes partitioned tables employing the MyISAM storage engine.
One issue might be
select MINUTE('2008-10-10 56:56:98') returns null, the reason is Minute function returns minute from time or datetime value, where as in your case date is varchar
MINUTE function returns in either date/datetime expression. Again, A partitioning key must be either an integer column or an expression that resolves to an
integer but inyour case it's VARCHAR
I want to partition a mysql table by datetime column. One day a partition.The create table scripts is like this:
CREATE TABLE raw_log_2011_4 (
id bigint(20) NOT NULL AUTO_INCREMENT,
logid char(16) NOT NULL,
tid char(16) NOT NULL,
reporterip char(46) DEFAULT NULL,
ftime datetime DEFAULT NULL,
KEY id (id)
) ENGINE=InnoDB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY hash (day(ftime)) partitions 31;
But when I select data of some day.It could not locate the partition.The select statement is like this:
explain partitions select * from raw_log_2011_4 where day(ftime) = 30;
when i use another statement,it could locate the partition,but I coluld not select data of some day.
explain partitions select * from raw_log_2011_4 where ftime = '2011-03-30';
Is there anyone tell me How I could select data of some day and make use of partition.Thanks!
Partitions by HASH is a very bad idea with datetime columns, because it cannot use partition pruning. From the MySQL docs:
Pruning can be used only on integer columns of tables partitioned by
HASH or KEY. For example, this query on table t4 cannot use pruning
because dob is a DATE column:
SELECT * FROM t4 WHERE dob >= '2001-04-14' AND dob <= '2005-10-15';
However, if the table stores year values in an INT column, then a
query having WHERE year_col >= 2001 AND year_col <= 2005 can be
pruned.
So you can store the value of TO_DAYS(DATE()) in an extra INTEGER column to use pruning.
Another option is to use RANGE partitioning:
CREATE TABLE raw_log_2011_4 (
id bigint(20) NOT NULL AUTO_INCREMENT,
logid char(16) NOT NULL,
tid char(16) NOT NULL,
reporterip char(46) DEFAULT NULL,
ftime datetime DEFAULT NULL,
KEY id (id)
) ENGINE=InnoDB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY RANGE( TO_DAYS(ftime) ) (
PARTITION p20110401 VALUES LESS THAN (TO_DAYS('2011-04-02')),
PARTITION p20110402 VALUES LESS THAN (TO_DAYS('2011-04-03')),
PARTITION p20110403 VALUES LESS THAN (TO_DAYS('2011-04-04')),
PARTITION p20110404 VALUES LESS THAN (TO_DAYS('2011-04-05')),
...
PARTITION p20110426 VALUES LESS THAN (TO_DAYS('2011-04-27')),
PARTITION p20110427 VALUES LESS THAN (TO_DAYS('2011-04-28')),
PARTITION p20110428 VALUES LESS THAN (TO_DAYS('2011-04-29')),
PARTITION p20110429 VALUES LESS THAN (TO_DAYS('2011-04-30')),
PARTITION future VALUES LESS THAN MAXVALUE
);
Now the following query will only use partition p20110403:
SELECT * FROM raw_log_2011_4 WHERE ftime = '2011-04-03';
Hi You are doing the wrong partition in definition of the table the table definition would like this:
CREATE TABLE raw_log_2011_4 (
id bigint(20) NOT NULL AUTO_INCREMENT,
logid char(16) NOT NULL,
tid char(16) NOT NULL,
reporterip char(46) DEFAULT NULL,
ftime datetime DEFAULT NULL,
KEY id (id)
) ENGINE=InnoDB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY hash (TO_DAYS(ftime)) partitions 31;
And your select command would be:
explain partitions
select * from raw_log_2011_4 where TO_DAYS(ftime) = '2011-03-30';
The above command would select all the date required, as if you use the TO_DAYS command as
mysql> SELECT TO_DAYS(950501);
-> 728779
mysql> SELECT TO_DAYS('2007-10-07');
-> 733321
Why to use the TO_DAYS AS The MySQL optimizer will recognize two date-based functions for partition pruning purposes:
1.TO_DAYS()
2.YEAR()
and this would solve your problem..
I just recently read a MySQL blog post relating to this, at http://dev.mysql.com/tech-resources/articles/mysql_55_partitioning.html.
Versions earlier than 5.1 required special gymnastics in order to do partitioning based on dates. The link above discusses it and shows examples.
Versions 5.5 and later allowed you to do direct partitioning using non-numeric values such as dates and strings.
Don't use CHAR, use VARCHAR. That will save a lot of space, hence decrease I/O, hence speed up queries. (Exception: If the column is really fixed length, then use CHAR. And it will probably be CHARACTER SET ascii.)
reporterip: (46) is unnecessarily big for an IP address, even IPv6. See My blog for further discussion, including how to shrink it to 16 bytes.
PARTITION BY RANGE(TO_DAYS(...)) as #Steyx suggested, but don't have more than about 50 partitions. The more partitions you have, the slower queries get, in spite of the "pruning". HASH partitioning is essentially useless.
More discussion of partitioning, especially the type you are looking at. That includes code for a sliding set of partitions over time.