My Table structure is
CREATE TABLE IF NOT EXISTS `billing_total_success` (
`bill_id` int(11) NOT NULL AUTO_INCREMENT,
`location` char(10) NOT NULL,
`circle` varchar(2) NOT NULL,
`amount` int(11) NOT NULL,
`reference_id` varchar(100) NOT NULL,
`source` varchar(100) NOT NULL,
`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`bill_id`),
KEY `location` (`location`),
KEY `soutime` (`source`,`time`),
KEY `circle` (`circle`,`source`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=80527470 ;
I need to partition this based on circle and subpartition on source.
Circle: Is 2 character from a set of 11 values ("AA","XB","BT"...)
Source: can be either "RNE"(sub partition 1) or "PR"(sub partition 2) or any other string(sub partition 3).
How do I do this partitioning?
Take a look at MySQL range partitioning. The closest you can get is to define a range less than "RNE", then a partition less than the next value which would occur after "RNE". Similar with PR, and then you can have a catch-all partition. The only issue here is that you need multiple partitions for before "RNE", the hole between "RNE" and "PR", and after "PR".
Perhaps if you could explain why you want to partition in this way, we could provide an alternative solution :)
Related
i have created partitions on pricing table. below is the alter statement.
ALTER TABLE `price_tbl`
PARTITION BY HASH(man_code)
PARTITIONS 87;
one partition consists of 435510 records. total records in price_tbl is 6 million.
EXPLAIN query showing only one partion is used for the query . Still the query takes 3-4 sec to execute. below is the query
EXPLAIN SELECT vrimg.image_cap_id,vm.man_name,vr.range_code,vr.range_name,vr.range_url, MIN(`finance_rental`) AS from_price, vd.der_id AS vehicle_id FROM `range_tbl` vr
LEFT JOIN `image_tbl` vrimg ON vr.man_code = vrimg.man_code AND vr.type_id = vrimg.type_id AND vr.range_code = vrimg.range_code
LEFT JOIN `manufacturer_tbl` vm ON vr.man_code = vm.man_code AND vr.type_id = vm.type_id
LEFT JOIN `derivative_tbl` vd ON vd.man_code=vm.man_code AND vd.type_id = vr.type_id AND vd.range_code=vr.range_code
LEFT JOIN `price_tbl` vp ON vp.vehicle_id = vd.der_id AND vd.type_id = vp.type_id AND vp.product_type_id=1 AND vp.maintenance_flag='N' AND vp.man_code=164
AND vp.initial_rentals_id =(SELECT rental_id FROM `rentals_tbl` WHERE rental_months='9')
AND vp.annual_mileage_id =(SELECT annual_mileage_id FROM `mileage_tbl` WHERE annual_mileage='8000')
WHERE vr.type_id = 1 AND vm.man_url = 'audi' AND vd.type_id IS NOT NULL GROUP BY vd.der_id
Result of EXPLAIN.
Same query without partitioning takes 3-4 sec.
Query with partitioning takes 2-3 sec.
how we can increase query performance as it is too slow yet.
attached create table structure.
price table - This consists 6 million records
CREATE TABLE `price_tbl` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`lender_id` bigint(20) DEFAULT NULL,
`type_id` bigint(20) NOT NULL,
`man_code` bigint(20) NOT NULL,
`vehicle_id` bigint(20) DEFAULT NULL,
`product_type_id` bigint(20) DEFAULT NULL,
`initial_rentals_id` bigint(20) DEFAULT NULL,
`term_id` bigint(20) DEFAULT NULL,
`annual_mileage_id` bigint(20) DEFAULT NULL,
`ref` varchar(255) DEFAULT NULL,
`maintenance_flag` enum('Y','N') DEFAULT NULL,
`finance_rental` decimal(20,2) DEFAULT NULL,
`monthly_rental` decimal(20,2) DEFAULT NULL,
`maintenance_payment` decimal(20,2) DEFAULT NULL,
`initial_payment` decimal(20,2) DEFAULT NULL,
`doc_fee` varchar(20) DEFAULT NULL,
PRIMARY KEY (`id`,`type_id`,`man_code`),
KEY `type_id` (`type_id`),
KEY `vehicle_id` (`vehicle_id`),
KEY `term_id` (`term_id`),
KEY `product_type_id` (`product_type_id`),
KEY `finance_rental` (`finance_rental`),
KEY `type_id_2` (`type_id`,`vehicle_id`),
KEY `maintenanace_idx` (`maintenance_flag`),
KEY `lender_idx` (`lender_id`),
KEY `initial_idx` (`initial_rentals_id`),
KEY `man_code_idx` (`man_code`)
) ENGINE=InnoDB AUTO_INCREMENT=5830708 DEFAULT CHARSET=latin1
/*!50100 PARTITION BY HASH (man_code)
PARTITIONS 87 */
derivative table - This consists 18k records.
CREATE TABLE `derivative_tbl` (
`type_id` bigint(20) DEFAULT NULL,
`der_cap_code` varchar(20) DEFAULT NULL,
`der_id` bigint(20) DEFAULT NULL,
`body_style_id` bigint(20) DEFAULT NULL,
`fuel_type_id` bigint(20) DEFAULT NULL,
`trans_id` bigint(20) DEFAULT NULL,
`man_code` bigint(20) DEFAULT NULL,
`range_code` bigint(20) DEFAULT NULL,
`model_code` bigint(20) DEFAULT NULL,
`der_name` varchar(255) DEFAULT NULL,
`der_url` varchar(255) DEFAULT NULL,
`der_intro_year` date DEFAULT NULL,
`der_disc_year` date DEFAULT NULL,
`der_last_spec_date` date DEFAULT NULL,
KEY `der_id` (`der_id`),
KEY `type_id` (`type_id`),
KEY `man_code` (`man_code`),
KEY `range_code` (`range_code`),
KEY `model_code` (`model_code`),
KEY `body_idx` (`body_style_id`),
KEY `capcodeidx` (`der_cap_code`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
range table - This consists 1k records
CREATE TABLE `range_tbl` (
`type_id` bigint(20) DEFAULT NULL,
`man_code` bigint(20) DEFAULT NULL,
`range_code` bigint(20) DEFAULT NULL,
`range_name` varchar(255) DEFAULT NULL,
`range_url` varchar(255) DEFAULT NULL,
KEY `range_code` (`range_code`),
KEY `type_id` (`type_id`),
KEY `man_code` (`man_code`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
PARTITION BY HASH is essentially useless if you are hoping for improved performance. BY RANGE is useful in a few use cases_.
In most situations, improvements in indexes are as good as trying to use partitioning.
Some likely problems:
No explicit PRIMARY KEY for InnoDB tables. Add a natural PK, if applicable, else an AUTO_INCREMENT.
No "composite" indexes -- they often provide a performance boost. Example: The LEFT JOIN between vr and vrimg involves 3 columns; a composite index on those 3 columns in the 'right' table will probably help performance.
Blind use of BIGINT when smaller datatypes would work. (This is an I/O issue when the table is big.)
Blind use of 255 in VARCHAR.
Consider whether most of the columns should be NOT NULL.
That query may be a victim of the "explode-implode" syndrome. This is where you do JOIN(s), which create a big intermediate table, followed by a GROUP BY to bring the row-count back down.
Don't use LEFT unless the 'right' table really is optional. (I see LEFT JOIN vd ... vd.type_id IS NOT NULL.)
Don't normalize "continuous" values (annual_mileage and rental_months). It is not really beneficial for "=" tests, and it severely hurts performance for "range" tests.
Same query without partitioning takes 3-4 sec. Query with partitioning takes 2-3 sec.
The indexes almost always need changing when switching between partitioning and non-partitioning. With the optimal indexes for each case, I predict that performance will be close to the same.
Indexes
These should help performance whether or not it is partitioned:
vm: (man_url)
vr: (man_code, type_id) -- either order
vd: (man_code, type_id, range_code, der_id)
-- `der_id` 4th, else in any order (covering)
vrimg: (man_code, type_id, range_code, image_cap_id)
-- `image_cap_id` 4th, else in any order (covering)
vp: (type_id, der_id, product_type_id, maintenance_flag,
initial_rentals, annual_mileage, man_code)
-- any order (covering)
A "covering" index is an extra boost, in that it can do all the work just in the index's BTree, without touching the data's BTree.
Implement a bunch of what I recommend, then come back (in another Question) for further tweaking.
Usually the "partition key" should be last in a composite index.
I'm having a problem with partition on a table in MySQL. I want to partition table reminders depend on quarter of year, because I think I can delete list of reminders was expired later.
Maybe, I should add field in primary key or unique key to partition work. Please help me this problem.
CREATE TABLE `reminders`
(
`id` bigint(20) NOT NULL,
`mer_reminder_id` varchar(255) UNIQUE NOT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NULL DEFAULT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
`type` varchar(32) DEFAULT NULL,
`priority` int(11) DEFAULT NULL,
`due` timestamp NULL DEFAULT NULL,
`title` varchar(255) DEFAULT NULL,
`description` varchar(255) DEFAULT NULL,
`template` varchar(32) DEFAULT NULL,
`action_id` bigint(20) NOT NULL,
`merchant_id` bigint(20) NOT NULL,
`expired` bool NULL DEFAULT NULL,
PRIMARY KEY (`id`, `created_at`)
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4
PARTITION BY RANGE ( UNIX_TIMESTAMP(created_at) ) (
PARTITION zpm_reminder_reminders_q1_2019 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-04-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q2_2019 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-07-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q3_2019 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-10-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q4_2019 VALUES LESS THAN ( UNIX_TIMESTAMP('2020-01-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q1_2020 VALUES LESS THAN ( UNIX_TIMESTAMP('2020-04-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q2_2020 VALUES LESS THAN ( UNIX_TIMESTAMP('2020-07-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q3_2020 VALUES LESS THAN ( UNIX_TIMESTAMP('2020-10-01 00:00:00') ),
PARTITION zpm_reminder_reminders_q4_2020 VALUES LESS THAN ( UNIX_TIMESTAMP('2021-01-01 00:00:00') )
);
Problem: #1503 - A UNIQUE INDEX must include all columns in the table's partitioning function
The error message is pretty self-explanatory: if your table is partitioned and has a unique key, every column used in the unique key must also be used as part of the partition. You are using UNIX_TIMESTAMP(created_at) for partitioning, but you have a unique key on mer_reminder_id, so this configuration isn't valid.
My advice: don't use MySQL partitions for this. They are extremely limiting, and are often buggy as well. If you need to discard old rows in this table, use DELETE with a condition -- unless your table is extremely large, you'll spend much more time maintaining the partitions than you will save on faster delete operations.
(Unrelatedly, I see a potential design issue in your table -- the primary key in your table should probably be id, not (id, created_at). Timestamps almost never belong in primary keys.)
I have table 'items'. 18 mln records:
CREATE TABLE IF NOT EXISTS `items` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`log_id` int(11) NOT NULL,
`res_id` int(11) NOT NULL,
`link` varchar(255) NOT NULL,
`title` text NOT NULL,
`content` text NOT NULL,
`n_date` varchar(255) NOT NULL,
`nd_date` int(11) NOT NULL,
`s_date` int(11) NOT NULL,
`not_date` date NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `link_2` (`link`),
KEY `log_id` (`log_id`),
KEY `res_id` (`res_id`),
KEY `now_date` (`not_date`),
KEY `sql_index` (`res_id`,`id`,`not_date`)
) ENGINE=Aria DEFAULT CHARSET=utf8 PAGE_CHECKSUM=0 AUTO_INCREMENT=18382133 ;
Trying to partition this table I created a mini copy of it and include column 'not_date' in primary and uniq keys:
CREATE TABLE IF NOT EXISTS `part_items` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`log_id` int(11) NOT NULL,
`res_id` int(11) NOT NULL,
`link` varchar(255) NOT NULL,
`title` text NOT NULL,
`content` text NOT NULL,
`n_date` varchar(255) NOT NULL,
`nd_date` int(11) NOT NULL,
`s_date` int(11) NOT NULL,
`not_date` date NOT NULL,
PRIMARY KEY (`not_date`,`id`),
UNIQUE KEY `link_2` (`not_date`,`link`),
KEY `log_id` (`log_id`),
KEY `res_id` (`res_id`),
KEY `now_date` (`not_date`),
KEY `sql_index` (`res_id`,`id`,`not_date`)
) ENGINE=Aria DEFAULT CHARSET=utf8 PAGE_CHECKSUM=0
/*!50100 PARTITION BY RANGE ( TO_DAYS(not_date))
(PARTITION p_1 VALUES LESS THAN (735963) ENGINE = Aria,
PARTITION p_2 VALUES LESS THAN (736694) ENGINE = Aria) */ AUTO_INCREMENT=18414661 ;
Then I run sql_query:
alter table `part_items` PARTITION BY RANGE( TO_DAYS(not_date) ) (
PARTITION p_1 VALUES LESS THAN( TO_DAYS('2014-12-31') ),
PARTITION p_2 VALUES LESS THAN( TO_DAYS('2016-12-31') )
);
Then I trying to select records that must de in p_1 and explain partitions show me that searching was only in p_1. But when I select records that must be in p_2 explain partitions show full-scan(p_1,p_2).
What wrong in my code?
Queries:
explain partitions SELECT * FROM `part_items` where content like '%k%' and not_date < '2014-05-12'
explain partitions SELECT * FROM `part_items` where content like '%k%' and not_date > '2015-01-01'
And one more question: Is it possible to partitioning views?
When PARTITIONing by some date function, there is a chance of an invalid date being provided. That would lead to NULL; such values are stored in the first partition.
This is an issue that has tripped up many a developer. The typical 'workaround' is to have the first partition empty so that the effort of looking in it is (usually) minimal. In your case:
PARTITION p_0 VALUES LESS THAN(0)
Partitioning on less than about 6 partitions is usually not worth the effort; will you be adding more partitions?
(Caveat: My advice comes from years of MyISAM/InnoDB partitioning; I don't know of Aria works differently. I suspect that partitioning is mostly handled at then Engine-independent layer.)
I have a table with million of rows and the frequency of growth will probably increase in future, so far about 4.3 million rows are added in a month, causing the database to slow down. I have already applied indexing but it's not really optimizing the speed. Is applying Partitioning to such data favorable?
Also how can I apply partitioning on a table with million of rows? I know it will look something like this
ALTER TABLE gpsloggs
PARTITION BY KEY(DeviceCode)
PARTITIONS 10;
The problem is I was Partitioning on DeviceCode which is not a primary key so partitioning isn't permissible.
DROP TABLE IF EXISTS `gpslogss`;
CREATE TABLE `gpslogss` (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`DeviceCode` varchar(255) DEFAULT NULL,
`Latitude` varchar(255) DEFAULT NULL,
`Longitude` varchar(255) DEFAULT NULL,
`Speed` double DEFAULT NULL,
`rowStamp` datetime DEFAULT NULL,
`Date` varchar(255) DEFAULT NULL,
`Time` varchar(255) DEFAULT NULL,
`AlarmCode` int(11) DEFAULT NULL,
PRIMARY KEY `Id` (`Id`) USING BTREE,
KEY `DeviceCode` (`DeviceCode`) USING BTREE
);
So I altered the table and made the table in a new database with 0 records this way and it worked fine
DROP TABLE IF EXISTS `gpslogss`;
CREATE TABLE `gpslogss` (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`DeviceCode` varchar(255) DEFAULT NULL,
`Latitude` varchar(255) DEFAULT NULL,
`Longitude` varchar(255) DEFAULT NULL,
`Speed` double DEFAULT NULL,
`rowStamp` datetime DEFAULT NULL,
`Date` varchar(255) DEFAULT NULL,
`Time` varchar(255) DEFAULT NULL,
`AlarmCode` int(11) DEFAULT NULL,
KEY `Id` (`Id`) USING BTREE,
KEY `DeviceCode` (`DeviceCode`) USING BTREE
);
PARTITION BY KEY(DeviceCode)
PARTITIONS 10;
How should I render the code so that I can apply partitioning to the table with million of rows? How should I drop keys and alter the table to apply partitioning without damaging data?
Short answer: Don't.
Long answer: PARTITION BY KEY does not provide any performance benefit (that I know of). And why else use PARTITION?
Other notes:
You should use InnoDB for virtually all tables.
InnoDB tables should have an explicit PRIMARY KEY.
There is a DATETIME datatype; don't use VARCHAR for date or time, and don't split them.
latitude and longitude are numeric; don't use VARCHAR. FLOAT is a likely candidate (precise enough to differentiate vehicles, but not people).
Your real question is about speed. Let's see the slow SELECTs and work backward from them. Adding PARTITIONing is rarely a solution to performance.
I tried to create a table with range partitioning. But it shows the following error:
A primary must include all columns in the table's partitioning
location.
This is my SQL statement:
CREATE TABLE `tbl_emp_confirmation` (
`fld_id` int(11) NOT NULL AUTO_INCREMENT,
`fldemp_id` varchar(100) DEFAULT NULL,
`fldempname` varchar(100) DEFAULT NULL,
`fldjoindate` varchar(100) DEFAULT NULL,
`fldconfirmdate` Date NOT NULL,
`fldresigndate` varchar(100) DEFAULT NULL,
`fldstatus` varchar(50) DEFAULT NULL,
`fldcon_status` varchar(100) DEFAULT NULL,
UNIQUE KEY `fld_id` (`fld_id`),
KEY `in_empconfirmation` (`fldemp_id`,`fldempname`,`fldjoindate`,`fldconfirmdate`)
) PARTITION BY RANGE ( Month(fldconfirmdate))
(PARTITION p_JAN VALUES LESS THAN (TO_DAYS('2011-01-01')),
PARTITION p_FEB VALUES LESS THAN (TO_DAYS('2011-02-01')),
PARTITION p_MAR VALUES LESS THAN (TO_DAYS('2011-03-01')),
PARTITION p_APR VALUES LESS THAN (TO_DAYS('2011-04-01')),
PARTITION p_MAY VALUES LESS THAN (TO_DAYS('2011-05-01')),
PARTITION p_MAX VALUES LESS THAN MAXVALUE );
You are partitioning data using fldconfirmdate, which is part of your PK, but not a part of your UNIQUE KEY fld_id.
This is extracted from the MySQL manual:
In other words, every unique key on the table must use every column in the table's partitioning expression.
Which means that, making fldconfirmdate to be a part of your UNIQUE KEY 'fld_id´ will solve the problem.
CREATE TABLE `tbl_emp_confirmation` (
`fld_id` int(11) NOT NULL AUTO_INCREMENT,
`fldemp_id` varchar(100) DEFAULT NULL,
`fldempname` varchar(100) DEFAULT NULL,
`fldjoindate` varchar(100) DEFAULT NULL,
`fldconfirmdate` Date NOT NULL,
`fldresigndate` varchar(100) DEFAULT NULL,
`fldstatus` varchar(50) DEFAULT NULL,
`fldcon_status` varchar(100) DEFAULT NULL,
UNIQUE KEY `fld_id` (`fld_id`, `fldconfirmdate`),
KEY `in_empconfirmation` (`fldemp_id`,`fldempname`,`fldjoindate`,`fldconfirmdate`)
) PARTITION BY RANGE ( Month(fldconfirmdate))
(PARTITION p_JAN VALUES LESS THAN (TO_DAYS('2011-01-01')),
PARTITION p_FEB VALUES LESS THAN (TO_DAYS('2011-02-01')),
PARTITION p_MAR VALUES LESS THAN (TO_DAYS('2011-03-01')),
PARTITION p_APR VALUES LESS THAN (TO_DAYS('2011-04-01')),
PARTITION p_MAY VALUES LESS THAN (TO_DAYS('2011-05-01')),
PARTITION p_MAX VALUES LESS THAN MAXVALUE );