SQL Partitioning of a very large table - mysql

Question

I'm trying to partition my very large MySQL table called companyScores (60million rows and 50 columns).
Basically, the table features companies (with the column varchar "company_idx" with unique IDs going from 0 to 10,000 companies) and their respective timestamp (with the column "timestamp") and scores "Scores" (with the column "Scores").
I'd like to include around 500 companies into each partition.
Please let me know if the following would do the job?
ALTER TABLE `companyScores`
PARTITION BY RANGE( company_idx ) (
PARTITION p0 VALUES LESS THAN (500),
PARTITION p1 VALUES LESS THAN (1000),
PARTITION p2 VALUES LESS THAN (1500),
PARTITION p3 VALUES LESS THAN (2000),
and so on...
);
Would the above work?
Also, can we easily insert new values into this database once it has been partitioned?

Answer 1

Would the above work?
No. For several reasons.
If company_idx is a varchar, you need to use RANGE COLUMNS. The RANGE partitioning only works on integers. If you try to use RANGE partitioning on a varchar, you get this error:
ERROR 1659 (HY000): Field 'company_idx' is of a not allowed type for this type of partitioning
Assuming you correct that, you have another problem:
Your partition clauses use integer values, not quoted string values. Those are different types, and the partitioning engine won't use them for defining partitions. If you try, you'll this this error:
ERROR 1654 (HY000): Partition column values of incorrect type
Assuming you correct that by quoting the numbers, you have another problem:
You list the partition for 500 before the string 1000, but the string '500' should come after the string '1000' lexically. RANGE or RANGE COLUMNS partitions must be declared in increasing order. If you try to do it in the order you have, you'll get this error:
ERROR 1493 (HY000): VALUES LESS THAN value must be strictly increasing for each partition
Assuming you correct the order, it works, but it might not do what you want:
CREATE TABLE `companyScores` (
`company_idx` varchar(10) NOT NULL,
PRIMARY KEY (`company_idx`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
/*!50500 PARTITION BY RANGE COLUMNS(company_idx)
(PARTITION p1 VALUES LESS THAN ('1000') ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN ('1500') ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN ('2000') ENGINE = InnoDB,
PARTITION p0 VALUES LESS THAN ('500') ENGINE = InnoDB) */
Now another question you asked:
Also, can we easily insert new values into this database once it has been partitioned?
If you insert a new value that isn't covered by the partitions you defined, you'll get this error:
mysql> insert into companyScores set company_idx = '700';
ERROR 1526 (HY000): Table has no partition for value from column_list
Why is that? You have a partition for company_idx less than 1000 right?
No. You have a partition for company_idx less than the string '1000'. You tried to insert the string '700', which is lexically greater than '500', as well as all the other partitions. Therefore it's beyond any of the partitions defined.
You could solve all of the above problems if you change your customer_idx to an integer column.