I have a database which use sequence number as its primary key. Other than there is a column called "date_time" which can be duplicated.
Now I need to make partitions by using date_time as follows.
ALTER TABLE data
PARTITION BY RANGE (TO_DAYS('date_time')) (
PARTITION p20220103 VALUES LESS THAN (TO_DAYS('2022-01-04 00:00:00')),
PARTITION p20220104 VALUES LESS THAN (TO_DAYS('2022-01-05 00:00:00')),
PARTITION p20220105 VALUES LESS THAN MAXVALUE
);
Since the date_time is not a primary key in data table, I couldn't create partitions.
ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the table's partitioning function (prefixed columns are not considered).
How should I create partitions without adding date_time as a primary key?
You cannot. The rule is simple, stated in https://dev.mysql.com/doc/refman/8.0/en/partitioning-limitations-partitioning-keys-unique-keys.html:
Every unique key on the table must use every column in the table's partitioning expression.
If the table has a primary key or unique key but that key does not include the column(s) in the partitioning expression, then it cannot enforce uniqueness when you insert a new row without checking every partition for duplicates.
The only way around this, to allow a column like your date_time to be the partitioning expression, is to define the table with no primary or unique key.
This has its own hazards. You may need a unique key so you can address rows individually to update or delete them. Also row-based replication becomes very inefficient if your table has no primary key.
This usually means you cannot partition the table by date_time, or even that you cannot partition the table at all. But this isn't always a bad thing. Partitioning doesn't necessarily give a great benefit. Partitioning can even cause more complexity, because you may have queries that would be bound to search every partition anyway.
Partitioning is not a cure-all, and frequently is a liability.
Related
I have a MySQL table (with data):
CREATE TABLE IF NOT EXISTS `lc6_words` (
`jp_wkey` BIGINT NOT NULL AUTO_INCREMENT,
`jp_word` varchar(255) NOT NULL,
`jp_fcharascii` INT NOT NULL,
`jp_word_occ` BIGINT NOT NULL DEFAULT 1,
UNIQUE(`jp_word`),
PRIMARY KEY (`jp_wkey`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
I want to add partitions in it (by altering):
ALTER TABLE lc6_words PARTITION BY RANGE COLUMNS(jp_fcharascii)(
PARTITION pw2486 VALUES LESS THAN (2486),
PARTITION pw2487 VALUES LESS THAN (2487),
PARTITION pw2488 VALUES LESS THAN (2488),
PARTITION pw2489 VALUES LESS THAN (2489),
PARTITION pwmax VALUES LESS THAN (MAXVALUE)
);
I'm getting the error:
Error Code: 1503. A PRIMARY KEY must include all columns in the
table's partitioning function
MySQL version: 5.7.19 / Win 10 64bit
Any way around to implement the partition keeping the table structure intact? Thanks in advance.
The error explains it pretty clearly.
https://dev.mysql.com/doc/refman/5.7/en/partitioning-limitations-partitioning-keys-unique-keys.html also says:
All columns used in the partitioning expression for a partitioned
table must be part of every unique key that the table may have.
In other words, every unique key on the table must use every column in
the table's partitioning expression.
That doc page goes on to show several examples of tables that may or may not be partitioned, or may be partitioned after some alteration.
Your table resembles one of the examples in the doc page that is shown as an example of an invalid partitioning request.
There are two workarounds, but both involve altering the table structure:
Add jp_fcharascii to your PRIMARY KEY and to the UNIQUE KEY on jp_word.
Remove both PRIMARY KEY and UNIQUE KEY constraints from the table (even if you keep the columns on which the constraints are defined).
But your condition in your question "keeping the table structure intact" precludes any alteration.
i've a large table with about 2Mln records, and i want to partition it.
I've the id column as PRIMARY AUTO_INCREMENT int (and it must to be always UNIQUE).
I've a column "theyear" int(4) and i want to partition BY RANGE from 2016 to 2050, because most of Query use a WHERE statement getting 1 year at time.
Making the partitioning i get an error saying that the "theyear" key must be with PRIMARY KEY, so i've edited the primary key doing a multicolumn key PRIMARY (id, theyear).
It's all OK, but my "id" columns isn't UNIQUE anymore, because it checks "theyear" columns too...
So if I insert:
INSERT INTO table (id, theyear) VALUES (1, 2016);
INSERT INTO table (id, theyear) VALUES (1, 2017);
it says NO ERROR, because the UNIQUE check both id and theyear.
How can implement partitioning without lose UNIQUE on "id" column?
Thanks.
Since your queries WILL use theyear in it, partitioning by RANGE will help with large data, with the proper INDEX.
Don't use KEY() partitionning, keep your existing primary key
INDEX IDX_theyear(theyear)
and
PARTITION BY RANGE (theyear) (
PARTITION p0 VALUES LESS THAN (2017),
PARTITION p1 VALUES LESS THAN (2018),
PARTITION p2 VALUES LESS THAN (2019),
PARTITION p3 VALUES LESS THAN (2020)
for example. You can add more partitions later, but harder. If your data is older, then obviously start with the oldest year
In mysql can I have a composite primary key composed of an auto increment and another field? Also, please critique my “mysql partitioning” logic
To explain further->
I have a query about MySQL partition.
I have to partition a table in MySQL, It has one primary key id.
I have to partition by date field(non-primary,duplicate entries).
Since we cannot partition on duplicate entries, i have created a composite key->(id,date).
How can i create partition in this composite key?
Thanks in Advance...
(This answer assumes InnoDB, not MyISAM. There are differences in the implementation of indexes that make some of my comments incorrect for MyISAM.)
In MySQL, a table's PRIMARY KEY can be composed of multiple fields, including an AUTO_INCREMENT.
The only requirement in MySQL for AUTO_INCREMENT is that it be the first column in some index. Let's look at this example of Posts, where there can be many posts for each user:
PRIMARY KEY(user_id, post_id),
INDEX(post_id)
where post_id is AUTO_INCREMENT, but you could benefit from "clustering" the data by user_id. This clustering would make it more efficient to do queries like
SELECT ... FROM Posts
WHERE user_id = 1234;
Back to your question...
The "partition key" does not have to be unique; so, I don't understant your "cannot partition on duplicate entries".
INDEX(id, date), if you also have PRIMARY KEY(id), is essentially useless. When looking up by id, the PRIMARY KEY(id) gives you perfect access; adding date to an index won't help. When looking up by date, but not id, (id, date) is useless since only the "left" part of a composite index can be used.
Perhaps you are leading to a non-partitioned table with
PRIMARY KEY(date, id),
INDEX(id)
to make date ranges efficient? (Note: partitioning won't help.)
Perhaps you will be doing
SELECT ... WHERE x = 123 AND date BETWEEN ...
In that case this is beneficial:
INDEX(x, date)
Only if you do this can we begin to discuss the utility of partitioning:
WHERE x BETWEEN ...
AND date BETWEEN ...
This needs a "two-dimensional" index, which sort of exists with SPATIAL.
See my discussion of partitioning where I list only 4 use cases for partitioning. It also links to an a discussion on how to use partitioning for 2D.
Bottom Line: You must not discuss partitioning without having a clear picture of what queries it might help. Provide them; then we can discuss further.
How MySQL create index for a partition table, Example if I create 5 hash by ID partitions:
Create 1 global index for all data and 5 partitions will use this index
Create 5 partitioned index with subdata in 5 partitioned tables
Create 5 index with all data in 5 partitioned tables
Thanks
There is no "global" index for a partitioned table in MySQL.
The only indexes you can put on a partitioned table ends up being separate indexes on each partition. Each partition is effectively an independent table.
HASH partitioning is virtually useless; do you have a particular use for which you think it might be beneficial?
Addenda...
The size of the index is similar to that of a table.
Since there are no "global" indexes, you cannot have a UNIQUE key unless it includes the column(s) of the "partition key". Nor can you use FOREIGN KEYs.
There is no type of index that spans more than one table.
Partitioned table in Mysql has only local indexes support.
What does that mean? Every partition of the table stores its own B-Tree for the indexes. This can slow down the process of search if you don't have partition key as part of the index. Also for unique key constraint, you need to add partition key as part of the unique key.
Compared to Mysql, Oracle has concept of Global indexes as well. Global Indexes are very hard to manage.
I am not too sure How helpful Mysql partitions would be if it has ignored Global indexes.
I have a hypothetical table with a primary key that is a BIGINT. Let's say my table grows very large and I have to partition and create different partitions by date range. What happens with primary key? Does that mean I can exceed the capacity of the BIGINT since there are more tables now? How does MySQL keep from assigning duplicate primary keys assuming a BIGINT set to auto increment a unique value?
Thanks in advance...
Partitioning doesn't create a new table. There is still one table with many partitions. Auto increment unique value keeps it functionality. Data grouped in partitions by date field that you choose to create partitions with.
Take a look at this : http://dev.mysql.com/tech-resources/articles/mysql_55_partitioning.html