MySQL slow request on partitionned table - mysql

Question

I am facing a complet mystery.
I have create a table to store meteorolocal data. I have one value per hour, since 1979, for every 0.25 latitude and longitude.
This brings me to have billions of lines in the database.
Following multiples advices, I partionnated the table.
I choosed to partitionnate by years. This is how it looks like :
CREATE TABLE `MyTable` (
`latitude_100` SMALLINT NOT NULL, -- Smallint is 2 bytes, where float is 4. So we take latitude * 100
`longitude_100` SMALLINT NOT NULL, -- Same logic here
`time` DATETIME NOT NULL,
`final` TINYINT UNSIGNED NOT NULL,
`value` DOUBLE NOT NULL,
PRIMARY KEY (`latitude_100` ASC, `longitude_100` ASC, `time` ASC)
)
PARTITION BY HASH(YEAR(time)) PARTITIONS 45 ; -- This will work until 2023 included
In order to test, I injected in the table data only from 2015 to 2021.
The problem :
All SELECT from this table are extremly long.
Even worst, they are sometime stupidly long.
For example :
SELECT time, latitude_100, longitude_100, value
FROM MyTable
WHERE latitude_100 BETWEEN 500 AND 2000
AND longitude_100 BETWEEN 11600 AND 12800 AND
YEAR(time) = 1990 ;
Remember that there is NO data for 1990. By looking into the right partition, MySQL should see it immeditaly isn't it ?
MySQL explain me that it will look in all partition, which I do not understand why :
EXPLAIN SELECT time, latitude_100, longitude_100, value
FROM MyTable
WHERE latitude_100 BETWEEN 500 AND 2000
AND longitude_100 BETWEEN 11600 AND 12800 AND
YEAR(time) = 1990 ;
# id, select_type, table, partitions, type, possible_keys, key, key_len, ref, rows, filtered, Extra
1, SIMPLE, MyTable, p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44, range, PRIMARY, PRIMARY, 4, , 118295536, 11.11, Using where
When I do
SELECT * FROM information_schema.partitions WHERE TABLE_SCHEMA='MySchema' AND TABLE_NAME = 'MyTable' AND PARTITION_NAME IS NOT NULL
I can see that only 6 partitions have data, all other are empty.
Last think I tried was to formulate the WHERE differently, to maybe take advantage of the index :
SELECT time, latitude_100, longitude_100, value
FROM MyTable
WHERE latitude_100 BETWEEN 500 AND 2000
AND longitude_100 BETWEEN 11600 AND 12800 AND
time BETWEEN "1990-01-01 00:00:00" AND "1990-12-31 23:00:00" AND
YEAR(time) = 1990 ;
But this does not accelerate the execution. Only the EXPLAIN is a bit different (but not in termes of partition reading) :
# id, select_type, table, partitions, type, possible_keys, key, key_len, ref, rows, filtered, Extra
1, SIMPLE, MyTable, p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44, range, PRIMARY, PRIMARY, 9, , 118295536, 1.23, Using where
What do I do wrong ?
Why MySQL does not want to cooperate with partitionning ?
Thank you very much !
[Edit]
On technical side, the database is hosted on AWS RDS. It is powered by a "db.t4g.large" instance and user MySQL 8.0.27

Answer 1

Do not use PARTITION BY HASH! HASH will fail to do any pruning when using a date range (as you have!). Simply put, the Optimizer is not smart enough to see that your range fits in a single partition. Furthermore, HASH may unnecessarily be lumping two different years into the same partition. Instead, use PARTITION BY RANGE.
I know that RANGE(TO_DAYS(time)) works; perhaps RANGE(YEAR(time)) may work, depending on what version of MySQL you are using; check the specifics.
Hour: With some date arithmetic, you can shrink a 5-byte DATETIME down to a 3-byte MEDIUMINT. (A suitable change to PARTITION BY RANGE would be needed.)
Not enough: Since you are testing with only 7 years of data, my Partitioning suggestion will help only by a factor of 7.
DOUBLE? What are you measuring? DOUBLE takes 8 bytes and gives you about 16 significant digits. Even FLOAT (4 bytes, 7 digits) is likely to be overkill. For temperature (°C), consider DECIMAL(2) or TINYINT (-128..+127) or DECIMAL(4,2); they are 1,1,2 bytes, respectively. Extremes recorded: -89..+57. Note: °F would need one more byte in any INT or DECIMAL encoding. (I would guess that an instrument too close to a volcano or wildfire would fail to transmit data if the temp exceeded 99°C.)
Shrinking the DOUBLE would shrink the dataset size by about 1/3 -- worth the effort.
If you will end up with about 400GB rows, datatype size is very important.
So, let's dig deeper... Please provide
Amount of RAM
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
Any other SELECTs that you are likely to run, including WHERE clauses other than exactly one year.
How much disk space did your 7 years take? If using MyISAM, I would expect about 1.2TB; if using InnoDB, 3TB.
The lat/lng ranges in the sample Select were relatively small. Is this typical? If so, we may be able to take advantage of it.
ENGINE -- Since this is, I assume, mostly a readonly dataset, it may be a rare case where MyISAM is better. See estimates above; multiple by 6 to get estimates for the 43 years.
Usage -- What will you do with the results of a SELECT like the one you have? If that is the 'only' query, then there are more compact ways to store the data. But they will be more complex to Insert and Select. However, the speed improvement may be worth it. I need to see the various Selects before advising further.