I have the following MySQL table that is currently partitioned.
I want to add sub-partitioning on the 'message_read' key - this is to optimize searches through partitions for un-read messages.
CREATE TABLE `messages` (
`pid` int(50) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
`userid1` int(10) unsigned NOT NULL COMMENT 'Message Sender',
`userid2` int(10) unsigned NOT NULL COMMENT 'Message Receiver',
`message` text NOT NULL COMMENT 'Actual Message',
`message_read` tinyint(1) NOT NULL DEFAULT '0' COMMENT '0=Not Read & 1=Read',
`datetime` datetime NOT NULL COMMENT 'Date Time',
PRIMARY KEY (`pid`,`datetime`),
KEY `userid1` (`userid1`),
KEY `userid2` (`userid2`),
KEY `message_read` (`message_read`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
/*!50100 PARTITION BY RANGE (TO_DAYS(`datetime`))
(PARTITION p0 VALUES LESS THAN (735234) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (735265) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (735293) ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN (735324) ENGINE = InnoDB,
PARTITION p4 VALUES LESS THAN (735354) ENGINE = InnoDB,
PARTITION p5 VALUES LESS THAN (735385) ENGINE = InnoDB,
PARTITION p6 VALUES LESS THAN (735415) ENGINE = InnoDB) */
Current partition is on datetime stamp. How could I add into sub-partitioning on 'message_read'?
thankyou
You'd need to add message_read to the primary key.
Then (You can use hash or key for sub-partitioning):
CREATE TABLE `messages` (
`pid` INT(50) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
`userid1` INT(10) UNSIGNED NOT NULL COMMENT 'Message Sender',
`userid2` INT(10) UNSIGNED NOT NULL COMMENT 'Message Receiver',
`message` TEXT NOT NULL COMMENT 'Actual Message',
`message_read` TINYINT(1) NOT NULL DEFAULT '0' COMMENT '0=Not Read & 1=Read',
`datetime` DATETIME NOT NULL COMMENT 'Date Time',
PRIMARY KEY (`pid`,`datetime`, message_read),
KEY `userid1` (`userid1`),
KEY `userid2` (`userid2`),
KEY `message_read` (`message_read`)
) ENGINE=INNODB DEFAULT CHARSET=utf8
PARTITION BY RANGE (TO_DAYS(`datetime`))
SUBPARTITION BY HASH (message_read)
(PARTITION p0 VALUES LESS THAN (735234)(SUBPARTITION s0, SUBPARTITION s1),
PARTITION p1 VALUES LESS THAN (735265)(SUBPARTITION s2, SUBPARTITION s3),
PARTITION p2 VALUES LESS THAN (735293)(SUBPARTITION s4, SUBPARTITION s5),
PARTITION p3 VALUES LESS THAN (735324)(SUBPARTITION s6, SUBPARTITION s7),
PARTITION p4 VALUES LESS THAN (735354)(SUBPARTITION s8, SUBPARTITION s9),
PARTITION p5 VALUES LESS THAN (735385)(SUBPARTITION s10, SUBPARTITION s11),
PARTITION p6 VALUES LESS THAN (735415)(SUBPARTITION s12, SUBPARTITION s13)) ;
Hope this was what you were looking for.
Thanks,
y
In this case - Since each partition is devided into two parts,
How will mysql devide the record to the subpartitions?
after all mysql don't know how many records will be entered into each partition/subpartition
Related
I am trying to create a table with partition with following way
create table transaction_demo
(
trx_id bigint not null primary key,
created_datetime datetime
) partition by range (YEAR(created_datetime)) (
partition p1 values less than (2019),
partition p2 values less than (2020),
partition p3 values less than (2021)
);
But it is not working. I am getting following error
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
How to fix this?
Then define primary key for each column which will be part of the partition.
CREATE TABLE transaction_demo (
trx_id int NOT NULL,
created_datetime datetime NOT NULL,
other_column_1 int NULL,
other_column_2 varchar(222) NULL,
PRIMARY KEY(trx_id,created_datetime)
)
PARTITION BY RANGE( YEAR(created_datetime) )(
PARTITION p1 VALUES LESS THAN (2019),
PARTITION p2 VALUES LESS THAN (2020),
PARTITION p3 VALUES LESS THAN (2021)
);
We have a ugly perl script that is used to parse csv data and populate this into MySQL. The original dev is long gone but now the script is failing and I'm trying to determine why.
MySQL Version 5.7.21
When running it we get this error.
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Table has no partition for value 217 [for Statement "INSERT INTO transaction ( case_qty, customer_pk, da_allocation, dist_customer_pk, dist_invoice_number, dist_item_pk, dist_manufacturer_pk, dist_quantity, dist_txn_date, distributor_pk, fb_item_pk, fb_manufacturer_pk, historical_hierarchy_pk, mfr_operator_va, mfr_rebateable_volume, min, original_rebate, period_time_pk, reporting_qty, reporting_uom, total_qty_purchased, total_vol, transaction_time_pk) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )" with ParamValues: 0=4, 1='19457', 2=0, 3='373809', 4='17755751', 5='106004', 6='75959', 7=4, 8='2018-01-19', 9='63', 10='227174', 11='130', 12='29546', 13=0, 14=0, 15='7888-928', 16=0, 17=217, 18=4, 19='CA', 20=41.3, 21=165.2, 22='217'] at load_usage.pl line 152
I'm trying to determine why exactly this MySQL statement is failing so I can find the underlying cause. All these columns are in the database and are spelled correctly.
Here is the columns we have in that table.
Output of show create table transaction:
CREATE TABLE `transaction` (
`pk` int(10) unsigned NOT NULL AUTO_INCREMENT,
`historical_hierarchy_pk` smallint(5) unsigned NOT NULL,
`dist_item_pk` int(10) unsigned NOT NULL,
`dist_manufacturer_pk` mediumint(8) unsigned NOT NULL,
`distributor_pk` smallint(5) unsigned NOT NULL,
`fb_item_pk` int(10) unsigned NOT NULL,
`fb_manufacturer_pk` mediumint(8) unsigned NOT NULL,
`customer_pk` mediumint(8) unsigned NOT NULL,
`dist_customer_pk` mediumint(8) unsigned NOT NULL,
`period_time_pk` smallint(5) unsigned NOT NULL,
`transaction_time_pk` smallint(5) unsigned NOT NULL,
`dist_txn_date` date NOT NULL,
`dist_quantity` decimal(7,2) NOT NULL DEFAULT '0.00',
`total_vol` decimal(9,2) NOT NULL DEFAULT '0.00',
`total_qty_purchased` decimal(10,4) NOT NULL DEFAULT '0.0000',
`reporting_qty` decimal(10,4) NOT NULL DEFAULT '0.0000',
`reporting_uom` char(2) NOT NULL DEFAULT '',
`case_qty` decimal(10,4) NOT NULL DEFAULT '0.0000',
`mfr_rebateable_volume` decimal(9,2) NOT NULL DEFAULT '0.00',
`mfr_operator_va` decimal(9,2) NOT NULL DEFAULT '0.00',
`dist_invoice_number` varchar(30) NOT NULL DEFAULT '',
`min` varchar(30) NOT NULL DEFAULT '',
`original_rebate` decimal(9,2) NOT NULL DEFAULT '0.00',
`da_allocation` decimal(9,2) NOT NULL DEFAULT '0.00',
PRIMARY KEY (`pk`,`period_time_pk`),
KEY `historical_hierarchy_pk_key` (`historical_hierarchy_pk`),
KEY `dist_item_pk_key` (`dist_item_pk`),
KEY `dist_manufacturer_pk_key` (`dist_manufacturer_pk`),
KEY `distributor_pk_key` (`distributor_pk`),
KEY `fb_item_pk_key` (`fb_item_pk`),
KEY `fb_manufacturer_pk_key` (`fb_manufacturer_pk`),
KEY `customer_pk_key` (`customer_pk`),
KEY `period_time_pk_key` (`period_time_pk`),
KEY `transaction_time_pk_key` (`transaction_time_pk`),
KEY `dist_customer_key` (`dist_customer_pk`)
) ENGINE=MyISAM AUTO_INCREMENT=351758733 DEFAULT CHARSET=latin1 COMMENT='$Revision$'
/*!50100 PARTITION BY RANGE (period_time_pk)
(PARTITION t0 VALUES LESS THAN (112) ENGINE = MyISAM,
PARTITION t1 VALUES LESS THAN (116) ENGINE = MyISAM,
PARTITION t2 VALUES LESS THAN (119) ENGINE = MyISAM,
PARTITION t3 VALUES LESS THAN (121) ENGINE = MyISAM,
PARTITION t4 VALUES LESS THAN (124) ENGINE = MyISAM,
PARTITION t5 VALUES LESS THAN (127) ENGINE = MyISAM,
PARTITION t6 VALUES LESS THAN (130) ENGINE = MyISAM,
PARTITION t7 VALUES LESS THAN (133) ENGINE = MyISAM,
PARTITION t8 VALUES LESS THAN (136) ENGINE = MyISAM,
PARTITION t9 VALUES LESS THAN (139) ENGINE = MyISAM,
PARTITION t10 VALUES LESS THAN (142) ENGINE = MyISAM,
PARTITION t11 VALUES LESS THAN (145) ENGINE = MyISAM,
PARTITION t12 VALUES LESS THAN (148) ENGINE = MyISAM,
PARTITION t13 VALUES LESS THAN (151) ENGINE = MyISAM,
PARTITION t14 VALUES LESS THAN (154) ENGINE = MyISAM,
PARTITION t15 VALUES LESS THAN (157) ENGINE = MyISAM,
PARTITION t16 VALUES LESS THAN (160) ENGINE = MyISAM,
PARTITION t17 VALUES LESS THAN (163) ENGINE = MyISAM,
PARTITION t18 VALUES LESS THAN (166) ENGINE = MyISAM,
PARTITION t19 VALUES LESS THAN (169) ENGINE = MyISAM,
PARTITION t20 VALUES LESS THAN (172) ENGINE = MyISAM,
PARTITION t21 VALUES LESS THAN (175) ENGINE = MyISAM,
PARTITION t22 VALUES LESS THAN (178) ENGINE = MyISAM,
PARTITION t23 VALUES LESS THAN (181) ENGINE = MyISAM,
PARTITION t24 VALUES LESS THAN (184) ENGINE = MyISAM,
PARTITION t25 VALUES LESS THAN (187) ENGINE = MyISAM,
PARTITION t26 VALUES LESS THAN (190) ENGINE = MyISAM,
PARTITION t27 VALUES LESS THAN (193) ENGINE = MyISAM,
PARTITION t28 VALUES LESS THAN (196) ENGINE = MyISAM,
PARTITION t29 VALUES LESS THAN (199) ENGINE = MyISAM,
PARTITION t30 VALUES LESS THAN (202) ENGINE = MyISAM,
PARTITION t31 VALUES LESS THAN (205) ENGINE = MyISAM,
PARTITION t32 VALUES LESS THAN (208) ENGINE = MyISAM,
PARTITION t33 VALUES LESS THAN (211) ENGINE = MyISAM,
PARTITION t34 VALUES LESS THAN (214) ENGINE = MyISAM,
PARTITION t35 VALUES LESS THAN (217) ENGINE = MyISAM) */ |
Your table uses RANGE partitioning and is not instructed what to do with values 217 and up.
Read up on what partitioning does and decide if you want to keep it. If so, you will by then also have figured out how you can solve your problem :) (Either by adding partitions or removing partitioning altogether).
The root cause of the error seems quite obvious: in the insert statement the value for period_time_pk is 217. If you check out the create table statement for the table, the table is partitioned by period_time_pk field and the last partition expression is:
PARTITION t35 VALUES LESS THAN (217) ENGINE = MyISAM
Therefore there is no partition defined that could hold the value 217. You need to add a new partition to hold this value.
I'm running MySQL 5.6 and have the following table:
CREATE TABLE `rawdata` (
`RawDataSeq` int(10) unsigned NOT NULL AUTO_INCREMENT,
`SpeciesSeq` int(10) unsigned DEFAULT NULL,
`DataSourceSeq` int(10) unsigned DEFAULT NULL,
`MeasurementSeq` int(10) unsigned DEFAULT NULL,
`RecordingDateTime` datetime DEFAULT NULL,
`RawSensorData` varchar(100) DEFAULT NULL,
`RawDataNumeric` decimal(36,18) DEFAULT NULL,
`RawDataString` varchar(100) DEFAULT NULL,
`RawDataDate` datetime DEFAULT NULL,
`RawDataTime` varchar(45) DEFAULT NULL,
`RawDataDateTime` datetime DEFAULT NULL,
`ActiveRecord` int(10) unsigned DEFAULT NULL,
`LastUpdateDate` datetime DEFAULT NULL,
`UserId` varchar(6) DEFAULT NULL,
`BadData` varchar(200) DEFAULT NULL,
PRIMARY KEY (`RawDataSeq`)
) ENGINE=MyISAM AUTO_INCREMENT=5676258 DEFAULT CHARSET=latin1
I'm trying to partition the table by 'RecordingDateTime' using the following command:
ALTER TABLE rawdata PARTITION BY RANGE (RecordingDateTime) (
PARTITION p0 VALUES LESS THAN (2013),
PARTITION p1 VALUES LESS THAN (2014),
PARTITION p2 VALUES LESS THAN (2015),
PARTITION p3 VALUES LESS THAN (UNIX_TIMESTAMP('2015-06-01 00:00:00')),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
I'm receiving the following error message:
Query: ALTER TABLE rawdata PARTITION BY RANGE (RecordingDateTime) ( PARTITION p0 VALUES LESS THAN (2013), PARTITION p1 VALUES LESS THAN...
Error Code: 1659
Field 'RecordingDateTime' is of a not allowed type for this type of partitioning
Any help is greatly appreciated.
Change to
PRIMARY KEY (`RawDataSeq`, RecordingDateTime)
ALTER TABLE rawdata PARTITION BY RANGE (TO_DAYS(RecordingDateTime)) (
PARTITION p_old VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION p2013 VALUES LESS THAN (TO_DAYS('2014-01-01')),
PARTITION p2014 VALUES LESS THAN (TO_DAYS('2015-01-01')),
PARTITION p2015 VALUES LESS THAN (TO_DAYS('2016-01-01')),
PARTITION future VALUES LESS THAN MAXVALUE
);
The code you need each December is found here.
I suspect there is a problem with the datetime value and the integer values you are using. One method is:
ALTER TABLE rawdata PARTITION BY RANGE (year(RecordingDateTime)) (
PARTITION p0 VALUES LESS THAN (2013),
PARTITION p1 VALUES LESS THAN (2014),
PARTITION p2 VALUES LESS THAN (2015),
PARTITION p3 VALUES LESS THAN MAXVALUE
);
If you don't store the RecordingDateTime as a timestamp, you can also try:
ALTER TABLE rawdata PARTITION BY RANGE (year(RecordingDateTime)*100 + month(RecordDateTime) (
PARTITION p0 VALUES LESS THAN (201300),
PARTITION p1 VALUES LESS THAN (201400),
PARTITION p2 VALUES LESS THAN (201500),
PARTITION p3 VALUES LESS THAN (201506),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
Otherwise, store the value as a timestamp and use timestamp ranges.
I have partitioned a table (because of an out of memory error - table got too big). I have partitioned it on a timestamp column as shown below:
CREATE TABLE test (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( unix_timestamp(dob) ) (
PARTITION p2012 VALUES LESS THAN (unix_timestamp('2013-01-01 00:00:00')),
PARTITION p2013 VALUES LESS THAN (unix_timestamp('2014-01-01 00:00:00')),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
I was hoping that the process of partitioning would also help in speeding up a couple of my queries whihc take a few hours to run; however, this type of partitioning doesn't seem to kick in and all partitions are still being used and scanned through for the queries. I have tried, and failed, with a couple more approaches:
1) Tried to use different range for the partitioning
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
However, that results in an error: Error Code: 1486. Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
2) Gave up on changing partitioning to be recognized by the query optimizer, and as suggested in MySQL's Doc - 18.5 Partition Selection tried specifying which partitions to use in the select statement instead:
select * from t2 partition (p2012)
But, that returns a syntax error Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(p2012) LIMIT 0, 1000' at line 1
Does anybody have any suggestions what else I could try to utilize table partitioning to optimize the queries?
You can use UNIX_TIMESTAMP() function. Example from MySQL docs:
CREATE TABLE quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-07-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-10-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-01-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-04-01 00:00:00') ),
PARTITION p6 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-07-01 00:00:00') ),
PARTITION p7 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-10-01 00:00:00') ),
PARTITION p8 VALUES LESS THAN ( UNIX_TIMESTAMP('2010-01-01 00:00:00') ),
PARTITION p9 VALUES LESS THAN (MAXVALUE)
);
You can find it in:
https://dev.mysql.com/doc/refman/5.7/en/partitioning-range.html.
You can do this if you use DATE or DATETIME instead of TIMESTAMP as the data type.
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob DATETIME NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
Using the partition-selection hint is only supported in MySQL 5.6 and later.
See http://dev.mysql.com/doc/refman/5.6/en/partitioning-selection.html
Note that the page of the manual is only for MySQL 5.6. If you try to click on MySQL 5.5 documentation link, it redirects you back to 5.6.
The wrong is that, This field dob is not unique key!
you can use this command:
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL,
unique 'dob' (dob)
)
or the table is exist:
alter table t2 add UNIQUE(dob)
You can try it!
The following CREATE TABLE statement to partition a table works as expected, without error.
CREATE TABLE `ox_data_archive_20120108` (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY LIST(to_days(datetime )) (
PARTITION `1Jan10` VALUES IN (to_days('2010-01-01') ),
PARTITION `2Jan10` VALUES IN (to_days('2010-01-02') ),
PARTITION `3Jan10` VALUES IN (to_days('2010-01-03') )
);
What I need to do is to create subpartitions based on date + zoneid. I tried the following:
CREATE TABLE mypart (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY LIST(to_days(datetime ))
SUBPARTITION BY KEY(zoneid) (
PARTITION `1Jan10` VALUES IN (to_days('2010-01-01') )
(Subpartition s1, Subpartition s2),
PARTITION `2Jan10` VALUES IN (to_days('2010-01-02') )
(Subpartition s3, Subpartition s4),
PARTITION `3Jan10` VALUES IN (to_days('2010-01-03') )
(Subpartition s5, Subpartition s6)
);
Inserting into this table:
INSERT INTO mypart VALUES (1, 2, 3, '2012-01-31 04:10:03');
results in the following error:
ERROR 1526 (HY000): Table has no partition for value 734898
My query expects to use the zoneid subpartition based on dates. Is it possible?
Contrary to your assertion that the first table works without error, inserting the sample data into it:
INSERT INTO `ox_data_archive_20120108` VALUES (1, 2, 3, '2012-01-31 04:10:03');
results in the same error as for the second table. The value given in the error (734898) happens to be the value for to_days('2012-01-31'). You get this error because you only have partitions for January 1st through 3rd, 2010. Both the day-of-month and the year for the sample data are outside the defined partitions. Instead of TO_DAYS (which returns the number of days from year 0 to the given date), you probably want DAYOFMONTH. Since each partition is contiguous, a RANGE partition seems more appropriate than a LIST.
Off topic, you only need to specify separate subpartition definitions when you want to set options for the subpartitions. Since you're not doing that, a SUBPARTITIONS 2 clause will do the same thing as your statement, but is simpler.
CREATE TABLE mypart (
`id` bigint(20) unsigned NOT NULL,
`creativeid` int unsigned NOT NULL,
`zoneid` int unsigned NOT NULL,
`datetime` datetime NOT NULL
) PARTITION BY RANGE(DAYOFMONTH(`datetime`))
SUBPARTITION BY KEY(zoneid)
SUBPARTITIONS 2 (
PARTITION `01` VALUES LESS THAN 2, -- Note: 0 is valid day-of-month
PARTITION `02` VALUES LESS THAN 3,
PARTITION `03` VALUES LESS THAN 4,
...
);