I am trying to merge partition created on daily bases based on timestamp but getting error.
I have created partition like this
$tomorrow = strtotime("tomorrow midnight");
$partition = date('dmy',$tomorrow);
ALTER TABLE `stats`
ADD PARTITION (
PARTITION p_$partition VALUES LESS THAN ($tomorrow)
)";
Then I do this to get daily partition
ALTER TABLE `stats`
PARTITION BY RANGE (`date`) (
PARTITION p_$partition VALUES LESS THAN ($tomorrow)
)
Now I need to merge all partition up till today into single partition doing this way
ALTER TABLE `stats` REORGANIZE PARTITION
p_190416,p_200416.....
INTO (
PARTITION p_$partition VALUES LESS THAN ($tomorrow)
);
But its showing me error as
Fatal error: Reorganize of range partitions cannot change total ranges except for last partition where it can extend the range in
Please see and suggest any possible way to do this without loosing data.
Related
MySQL 5.7
Question -
Can I make a table partition for every data that does not fit to any of predefined partitions? Or is it against the purpose of the partitioning concept?
for example,
ALTER TABLE sample_table PARTITION BY LIST COLUMNS(GROUP_CODE)(
PARTITION A_001 IN ('A001'),
PARTITION B_101 VALUES IN ('B101'),
PARTITION B_102 VALUES IN ('B102'),
PARTITION B_202 VALUES IN ('B202'),
PARTITION C_101 IN ('C101'),
PARTITION C_201 IN ('C201'),
PARTITION D_000 IN ('D000'),
);
If I make partitions as such, 'G525' won't be allowed to added to the table. If the partitioning key was integer I can go for the RANGE, but is it possible with the LIST?
I'm trying to partition my very large MySQL table called companyScores (60million rows and 50 columns).
Basically, the table features companies (with the column varchar "company_idx" with unique IDs going from 0 to 10,000 companies) and their respective timestamp (with the column "timestamp") and scores "Scores" (with the column "Scores").
I'd like to include around 500 companies into each partition.
Please let me know if the following would do the job?
ALTER TABLE `companyScores`
PARTITION BY RANGE( company_idx ) (
PARTITION p0 VALUES LESS THAN (500),
PARTITION p1 VALUES LESS THAN (1000),
PARTITION p2 VALUES LESS THAN (1500),
PARTITION p3 VALUES LESS THAN (2000),
and so on...
);
Would the above work?
Also, can we easily insert new values into this database once it has been partitioned?
Would the above work?
No. For several reasons.
If company_idx is a varchar, you need to use RANGE COLUMNS. The RANGE partitioning only works on integers. If you try to use RANGE partitioning on a varchar, you get this error:
ERROR 1659 (HY000): Field 'company_idx' is of a not allowed type for this type of partitioning
Assuming you correct that, you have another problem:
Your partition clauses use integer values, not quoted string values. Those are different types, and the partitioning engine won't use them for defining partitions. If you try, you'll this this error:
ERROR 1654 (HY000): Partition column values of incorrect type
Assuming you correct that by quoting the numbers, you have another problem:
You list the partition for 500 before the string 1000, but the string '500' should come after the string '1000' lexically. RANGE or RANGE COLUMNS partitions must be declared in increasing order. If you try to do it in the order you have, you'll get this error:
ERROR 1493 (HY000): VALUES LESS THAN value must be strictly increasing for each partition
Assuming you correct the order, it works, but it might not do what you want:
CREATE TABLE `companyScores` (
`company_idx` varchar(10) NOT NULL,
PRIMARY KEY (`company_idx`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
/*!50500 PARTITION BY RANGE COLUMNS(company_idx)
(PARTITION p1 VALUES LESS THAN ('1000') ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN ('1500') ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN ('2000') ENGINE = InnoDB,
PARTITION p0 VALUES LESS THAN ('500') ENGINE = InnoDB) */
Now another question you asked:
Also, can we easily insert new values into this database once it has been partitioned?
If you insert a new value that isn't covered by the partitions you defined, you'll get this error:
mysql> insert into companyScores set company_idx = '700';
ERROR 1526 (HY000): Table has no partition for value from column_list
Why is that? You have a partition for company_idx less than 1000 right?
No. You have a partition for company_idx less than the string '1000'. You tried to insert the string '700', which is lexically greater than '500', as well as all the other partitions. Therefore it's beyond any of the partitions defined.
You could solve all of the above problems if you change your customer_idx to an integer column.
I have a table which will grow large over time, moreover I need only small amount of data say last 7 days.
I want to configure it such that the data of 7 days goes in one partition, and then in next. This way I would keep only two partitions and archive others.
I read about MySQL partitions here but the way in article to create partitions is that we specify all partitions while creating table only.
I am not sure is this the best way to do it where we add partitioning logic for long time.
Any ideas?
Unfortunately, it'll be a fairly manual process. Your best bet is to create the partitions, week by week ahead of time, then have a job that runs periodically to archive the old data into the 'catchall' partition.
e.g. with:
PARTITION BY RANGE ( TO_DAYS(date) ) (
PARTITION pmin VALUES LESS THAN ( TO_DAYS('2016-10-02 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( TO_DAYS('2016-10-09 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( TO_DAYS('2016-10-16 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( TO_DAYS('2016-10-23 00:00:00') ),
PARTITION pmax VALUES LESS THAN (MAXVALUE)
);
There's no real harm having a few empty partitions sitting there with higher dates then doing a 'shift' once a week. It'll be fast enough as long as when you change the partitioning definition, the data window shifts by the partition size.
Your job would do something like
ALTER TABLE x REORGANIZE PARTITION pmin,p1 INTO (
PARTITION pmin VALUES LESS THAN ('2016-10-09 00:00:00')
);
ALTER TABLE x
ADD PARTITION px VALUES LESS THAN ( TO_DAYS('2016-10-30 00:00:00') )
);
There is no "automatic" partition management in MySQL. We have to run some specific SQL statements to add and drop partitions from a partitioned table.
We automated the task with a cron job which runs a MySQL PROCEDURE we wrote to drop (swap out) old partitions, and another PROCEDURE to add new partitions. The procedures are specific to a particular table.
Our table is partitioned by RANGE on a TIMESTAMP column. The partition expression is like UNIX_TIMESTAMP(col).
To add a new partition, we reorganize the MAXVALUE partition, which is always (or should always be) empty, so the operation is very quick. We dynamically prepare and execute a statement of the form:
ALTER TABLE ourtable REORGANIZE PARTITION pmax
INTO ( PARTITION pn_name VALUES LESS THAN (UNIX_TIMESTAMP(pn_date))
, PARTITION pmax VALUES LESS THAN MAXVALUE)
To get a new date value for the new partition (pn_name), we take the partition_description value from the second to last partition (the last partition is the MAXVALUE partition), and add 7 days to it to get the pn_date string to use. We use that same value to generate the pn_name for the new partition. (We name the partitions following a pattern like this: p20161030 based on the date value in the partition_description e.g. UNIX_TIMESTAMP('2016-10-30').
(This information is obtained from a fairly involved query with a couple of references to information_schema.partitions view.
With the other procedure to drop old partitions, we actually "swap out" the old partition to an archive table. (The archive table is later backed up, and dropped by a different task.)
The procedure basically runs a series of statements like this:
DROP TABLE IF EXISTS `_et` ;
CREATE TABLE `_et` LIKE `rdg_point_value` ;
ALTER TABLE `_et` REMOVE PARTITIONING ;
ALTER TABLE `ourtable` EXCHANGE PARTITION `oldest_partition` WITH TABLE `_et` ;
ALTER TABLE `ourtable` DROP PARTITION `oldest_partition` ;
RENAME TABLE `et` TO `archive_oldest_partition` ;
(I wish there was a cleaner way to create a new un-partitioned table, in a single statement, such as a a CREATE TABLE ... LIKE ... WITHOUT PARTITIONING, but absent that, we settled on the two separate statements.)
Just dropping the oldest partition would be a simpler process.
To obtain information about the oldest partition, our query is probably overkill. But it's where most of the "magic" happens. Just to give you an idea of what that query looks like...
FROM information_schema.partitions p1
JOIN information_schema.partitions px
ON px.table_schema = 'ourdatabase'
AND px.table_name = 'ourtable'
AND px.partition_method = 'RANGE'
AND px.partition_expression = 'UNIX_TIMESTAMP(ourcol)'
AND px.partition_description = 'MAXVALUE'
WHERE p1.table_schema = 'ourdatabase'
AND p1.table_name = 'ourtable'
AND p1.partition_method = 'RANGE'
AND p1.partition_expression = 'UNIX_TIMESTAMP(ourcol)'
AND p1.partition_description <> 'MAXVALUE'
AND p1.partition_description + 0 <= UNIX_TIMESTAMP(DATE(NOW()) + INTERVAL -187 DAY)
AND p1.partition_ordinal_position = 1
You could probably get away with a simpler query. (Our query is designed to only return the "oldest" partition only if all of the timestamp values in it are at least six months old, and only if there is a MAXVALUE partition defined.
Each of the procedures use the current date to see if "its time" to add or drop a partition. (The amount of time forward and back is hardcoded into the queries in the procedure... the query returns 0 rows if its not time yet.
The procedures only need to be executed once per week, and we designed them so that any "extra" runs won't add or drop partitions outside of the specified time ranges.
We have the procedures scheduled to execute every day, and on most days, the procedure runs a query which returns zero rows, and exits. Only when the query returns a row is there any work to do.
MySQL throwing error while creating partitions on table.
Error Code : 1486
Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed.
I have tried following query :
alter table test.tbl1
partition by range(unix_timestamp(sys_time))
(
PARTITION p20151001 VALUES LESS THAN (unix_timestamp('2015-10-01')),
PARTITION p20151101 VALUES LESS THAN (unix_timestamp('2015-11-01')),
PARTITION p20151201 VALUES LESS THAN (unix_timestamp('2015-12-01')),
PARTITION p20160101 VALUES LESS THAN (unix_timestamp('2016-01-01')),
PARTITION p20160201 VALUES LESS THAN (unix_timestamp('2016-02-01')),
PARTITION p20160301 VALUES LESS THAN (unix_timestamp('2016-03-01'))
);
How can I round this problem.
Thanks in Advance
Reading here it may be possible that you are using MYSQL 5.1:
https://dev.mysql.com/tech-resources/articles/mysql_55_partitioning.html
Another pain point in MySQL 5.1 is the handling of date columns. You
can't use them directly, but you need to convert such columns using
either YEAR or TO_DAYS
If your column sys_time is a DATETIME, you dont need to specify the timestamp in order to partition it, you just need to do TO_DAYS, since you're not doing it by year:
alter table test.tbl1
partition by range (TO_DAYS(sys_time))
(
PARTITION p20151001 VALUES LESS THAN (TO_DAYS('2015-10-01')),
PARTITION p20151101 VALUES LESS THAN (TO_DAYS('2015-11-01')),
PARTITION p20151201 VALUES LESS THAN (TO_DAYS('2015-12-01')),
PARTITION p20160101 VALUES LESS THAN (TO_DAYS('2016-01-01')),
PARTITION p20160201 VALUES LESS THAN (TO_DAYS('2016-02-01')),
PARTITION p20160301 VALUES LESS THAN (TO_DAYS('2016-03-01'))
);
if sys_time is a TIMESTAMP then you dont need to convert your timestamp to a timestamp, I have taken that out of the range parameter:
alter table test.tbl1
partition by range(sys_time)
(
PARTITION p20151001 VALUES LESS THAN (unix_timestamp('2015-10-01')),
PARTITION p20151101 VALUES LESS THAN (unix_timestamp('2015-11-01')),
PARTITION p20151201 VALUES LESS THAN (unix_timestamp('2015-12-01')),
PARTITION p20160101 VALUES LESS THAN (unix_timestamp('2016-01-01')),
PARTITION p20160201 VALUES LESS THAN (unix_timestamp('2016-02-01')),
PARTITION p20160301 VALUES LESS THAN (unix_timestamp('2016-03-01'))
);
I have a table named edr on mysql 5.1.6* version. I have partitioned the table using alter -
ALTER TABLE edr PARTITION BY RANGE (TO_DAYS(eventDate))
(
PARTITION apr25 VALUES LESS THAN (TO_DAYS('2014-04-26')),
PARTITION apr26_30 VALUES LESS THAN (TO_DAYS('2014-05-01')),
PARTITION may01_05 VALUES LESS THAN (TO_DAYS('2014-05-06')),
PARTITION may06_10 VALUES LESS THAN (TO_DAYS('2014-05-11')),
PARTITION may11_15 VALUES LESS THAN (TO_DAYS('2014-05-16')),
PARTITION may16_20 VALUES LESS THAN (TO_DAYS('2014-05-21')),
PARTITION may21_25 VALUES LESS THAN (TO_DAYS('2014-05-26')),
PARTITION may26_31 VALUES LESS THAN (TO_DAYS('2014-06-01')),
PARTITION june01_05 VALUES LESS THAN (TO_DAYS('2014-06-06')),
PARTITION june06_10 VALUES LESS THAN (TO_DAYS('2014-06-11')),
PARTITION june11_15 VALUES LESS THAN (TO_DAYS('2014-06-16')));
now when I am running any query for example:
explain partitions select count(*) from edr where eventdate > '2014-05-21';
it gives me output for partitions as - apr25,may21_25, may26_31, jun01_05,jun_06_10,jun11_15.
Here in partition apr25 there is no record for such where condition.
please let me know is any thing wrong in above query or its a partition problem.
It is MySQL bug: explained here.
Try to create a first partition that contains values less than (0)
PARTITION unused VALUES LESS THAN (0);