Basic MySQL query is pretty slow. Am I indexing my tables properly? - mysql

Question

I'm currently in the process of optimizing a very large MySQL database, around which I'm building a web-based query interface.
The database will have two tables. The first table is already optimized (I believe), and contains information regarding 950 meteorological data observation stations across the US:
Description for: stations (950 records)
+-----------+------------+--------+-------+---------+----------------+
|Field |Type | NULL |KEY | Default | Extra |
+-----------+------------+--------+-------+---------+----------------+
|id |INT |NO |PRI |NULL |auto_increment |
|stationID |char(4) |NO |PRI |NULL | |
|name |varchar(16) |YES | |NULL | |
|state |char(2) |YES |MUL |NULL | |
|lat |float(6,2) |YES | |NULL | |
|lon |float(6,2) |YES | |NULL | |
|elev |INT |YES | |NULL | |
+-----------+------------+--------+-------+---------+----------------+
The other table contains observations collected at these stations from 2014 through 2017 (constructed, not optimized):
Description for: metar_records (359786049 records)
+-----------+------------+--------+-------+---------+----------------+
|Field |Type | NULL |KEY | Default | Extra |
+-----------+------------+--------+-------+---------+----------------+
|auto_id |INT |NO |PRI |NULL |auto_increment |
|stationID |char(4) |NO |MUL |0 | |
|zdatetime |datetime |NO | |NULL | |
|ldatetime |datetime |NO | |NULL | |
|temp |tinyint(4) |YES | |NULL | |
|dew |tinyint(4) |YES | |NULL | |
|wspd |tinyint(3) |YES | |NULL | #unsigned |
|wdir |tinyint(3) |YES | |NULL | #unsigned |
|wgust |tinyint(3) |YES | |NULL | #unsigned |
|VRB |char(3) |YES | |NULL | |
+-----------+------------+--------+-------+---------+----------------+
where stationID is the field upon which the two tables are related. The metar_records has a unique index on ('stationID', 'zdatetime'). A list of metar_records table indexes:
+-------------+--------+---------+------------+-----------+-----------+----------+
|Table |Non_UNQ |Key_name |Seq_in_index|Column_name|Cardinality|Index_type|
+-------------+--------+---------+------------+-----------+-----------+----------+
|metar_records|0 |PRIMARY |1 |auto_id |358374698 |BTREE |
|metar_records|0 |sz_date |1 |stationID |820079 |BTREE |
|metar_records|0 |sz_date |2 |zdatetime |358374698 |BTREE |
|metar_records|1 |stationID|1 |stationID |598288 |BTREE |
+-------------+--------+---------+------------+-----------+-----------+----------+
Here's where I'm really confused: I also have a test table (called metar_test), which is identical to metar_records aside from having no auto_increment field, and has no indexes whatsoever. Execution of SELECT COUNT(*) FROM metar_test; lasts 0.02 seconds at most, whereas SELECT COUNT(*) FROM metar_records; takes roughly 1 minute and 18 seconds to complete.
I understand that having a table this large will result in some long query times, but metar_records is only 3.36 times larger than metar_test -- why is there such a large discrepancy between the SELECT COUNT(*) ... queries for the two tables? I'm not particularly well versed in data storage, but this difference seems unexpectedly large to me.
How can I improve my indexing to optimize the large table size? Is it possible to reduce the query duration from here?

Answer 1

You can try:
select count(stationID)
from metar_records
This will make the query optimizer to use the index of stationID and therefore reading less data as the count(*) which reads through full data.

Answer 2

I would reconstruct your tables in such way.
Stations.
Id as auto inc.
StationId char (4) unique
Rest...
Metar_records
Id as auto inc
StationId referencing stations.id
Rest...
This way your key length are way smaller and numeric. Will boost up your performance.

Answer 3

You probably have the "Query cache" turned on. That makes it so that running exactly the same query a second time is very fast. To properly time a query, do this twice and take the second timing:
SELECT SQL_NO_CACHE ...
COUNT(*) is the usual pattern for counting rows. COUNT(col) is slower because it needs to check each col for being NOT NULL.
You have 3 INDEXes on your big table; you need only one:
PRIMARY KEY(stationID, zdatetime)
And, by clustering that way, several likely queries will run faster.
Please use SHOW CREATE TABLE; it is more descriptive than DESCRIBE.
You should use ENGINE=InnoDB, not ENGINE=MyISAM (see SHOW CREATE TABLE).
SELECT COUNT(*) ... is not a very common query; you should not take much stock in how fast it runs.
PARTITIONing is not likely to help performance. Let's see more of your queries -- to double check my claim. MySQL has no parallel processing, even for PARTITIONed tables.
Also toss the id AUTO_INCREMENT from the Stations table; instead, have PRIMARY KEY(stationID).