MySQL Partitioning and Automatic Movement of Rows - mysql

Question

I have a table with ~6M rows that is extracting around ~20,000-30,000 rows per query with index optimization. However, as a lot of people are extracting these rows consecutively (every 30 seconds or so) the site will often time out for people.
I recently migrated the database to a 3-server MySQL cluster with a huge amount of RAM (512GB per server) and the performance haven't improved a lot.
I was wondering if partioning would be the best way to proceed to improve performance. As I have absolutely no experience with partioning I thought I would ask here.
My question is, all of these rows have a column that will either have the value 0, 1, 2 or 3.
Would it be possible somehow to place all the rows with value 1 in a certain column on one partition, and all rows with value 2 in a column in another one? And would they move automatically based on the value being updated in the primary table? And most importantly, could it help out with performance as it would only have to look through finding 1 row in 20,000-30,000 instead of 6,000,000

Answer 1

Yes, MySQL supports partitioning. You can define the partitions pretty well, like:
CREATE TABLE MyTable (
id INT AUTO_INCREMENT PRIMARY KEY,
somestuff INT,
otherstuff VARCHAR(100),
KEY (somestuff)
) PARTITION BY HASH(id) PARTITIONS 4;
INSERT INTO MyTable () VALUES (), (), (), ();
You can verify how many rows are in each partition after this:
SELECT PARTITION_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME='MyTable';
+----------------+------------+
| PARTITION_NAME | TABLE_ROWS |
+----------------+------------+
| p0 | 1 |
| p1 | 1 |
| p2 | 1 |
| p3 | 1 |
+----------------+------------+
However, there are two things that trip people up when they try to use partitioning in MySQL:
First, as https://dev.mysql.com/doc/refman/5.7/en/partitioning-limitations-partitioning-keys-unique-keys.html says:
every unique key on the table must use every column in the table's partitioning expression.
This means if you want to partition by somestuff in the example above, you can't. That would fail the requirement that primary key include the column named in the partition expression.
ALTER TABLE MyTable PARTITION BY HASH(somestuff) PARTITIONS 4;
ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the table's partitioning function
You can get around this by removing any primary key or unique key constraints from your table, but that leaves you with kind of a malformed table.
Second, partitioning speeds up queries only if you can take advantage of partition pruning, and this happens only if your query conditions include the column used in the partition expression.
mysql> EXPLAIN PARTITIONS SELECT * FROM MyTable WHERE SomeStuff = 3;
+----+-------------+---------+-------------+------+---------------+-----------+---------+-------+------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+-------------+------+---------------+-----------+---------+-------+------+-------+
| 1 | SIMPLE | MyTable | p0,p1,p2,p3 | ref | somestuff | somestuff | 5 | const | 4 | NULL |
+----+-------------+---------+-------------+------+---------------+-----------+---------+-------+------+-------+
Note this says it will need to scan partitions p0,p1,p2,p3 — i.e. the whole table. There is no partition pruning, therefore no performance improvement because it is not reducing the number of rows examined.
If you do search for a specific value in the column used in the partitioning expression, you can see that MySQL is able to reduce the number of partitions it scans:
mysql> EXPLAIN PARTITIONS SELECT * FROM MyTable WHERE id = 3;
+----+-------------+---------+------------+-------+---------------+---------+---------+-------+------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+------------+-------+---------------+---------+---------+-------+------+-------+
| 1 | SIMPLE | MyTable | p3 | const | PRIMARY | PRIMARY | 4 | const | 1 | NULL |
+----+-------------+---------+------------+-------+---------------+---------+---------+-------+------+-------+
Partitioning can help a lot in very specific circumstances, but partitioning isn't as versatile as most people think.
In most cases, it's better to define more specific indexes in your table to support the queries you need to run.