MySQL database slowing down - mysql

Question

I need some help figuring out a performance issue. A database containing a single table with a growing number of METARs (aviation weather reports) is slowing down after about 8 million records are present. This despite indexes being in use. Performance can be recovered by rebuilding indexes, but that's really slow and takes the database offline, so I've resorted to just dropping the table and recreating it (losing the last few weeks of data).
The behaviour is the same whether a query is run trying to retrieve an actual metar, or whether a simple select count(*) is executed.
The table creation syntax is as follows:
CREATE TABLE `metars` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`tstamp` timestamp NULL DEFAULT NULL,
`metar` varchar(255) DEFAULT NULL,
`icao` char(7) DEFAULT NULL,
`qnh` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `timestamp` (`tstamp`),
KEY `icao` (`icao`),
KEY `qnh` (`qnh`),
KEY `metar` (`metar`)
) ENGINE=InnoDB AUTO_INCREMENT=812803050 DEFAULT CHARSET=latin1;
Up to about 8 million records, a select count(*) returns in about 500ms. Then it gradually increases, currently again at 14 million records, the count takes between 3 and 30 seconds. I was surprised to see that when explaining the count query, it's using the timestamp as an index, not the primary key. Using the primary key this should be a matter of just a few ms to return the number of records:
mysql> explain select count(*) from metars;
+----+-------------+--------+-------+---------------+-----------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+-------+---------------+-----------+---------+------+----------+-------------+
| 1 | SIMPLE | metars | index | NULL | timestamp | 5 | NULL | 14693048 | Using index |
+----+-------------+--------+-------+---------------+-----------+---------+------+----------+-------------+
1 row in set (0.00 sec)
Forcing it to use the primary index is even slower:
mysql> select count(*) from metars use index(PRIMARY);
+----------+
| count(*) |
+----------+
| 14572329 |
+----------+
1 row in set (37.87 sec)
Oddly, the typical use case query is to get the weather for an airport nearest to a specific point in time which continues to perform very well, despite being more complex than a simple count:
mysql> SELECT qnh, metar from metars WHERE icao like 'KLAX' ORDER BY ABS(TIMEDIFF(tstamp, STR_TO_DATE('2019-10-10 00:00:00', '%Y-%m-%d %H:%i:%s'))) LIMIT 0,1;
+------+-----------------------------------------------------------------------------------------+
| qnh | metar |
+------+-----------------------------------------------------------------------------------------+
| 2980 | KLAX 092353Z 25012KT 10SM FEW015 20/14 A2980 RMK AO2 SLP091 T02000139 10228 20200 56007 |
+------+-----------------------------------------------------------------------------------------+
1 row in set (0.01 sec)
What am I doing wrong here?

Answer 1

InnoDB performs a plain COUNT(*) by traversing some index. It prefers the smallest index because that will require touching the least number of blocks.
The PRIMARY KEY is clustered with the data, so that index is actually the biggest.
What version are you using? TIMESTAMP changed at some point. Perhaps that explains why tstamp is used instead of qnh.
If you are purging old data by using DELETE, see http://mysql.rjweb.org/doc.php/partitionmaint for a faster way.
I assume the data is static; that is it is never UPDATEd? Consider building and maintaining a summary table, perhaps indexed by date. This could have various counts for each day. Then a fetch from that table would be much faster than hitting the raw data. More: http://mysql.rjweb.org/doc.php/summarytables
How many rows for KLAX? That query must fetch all of them in order to convert the timestamp before doing the LIMIT. If you had INDEX(icao, tstamp), you could find the next before or after a given time even faster.