Weird behavior of SELECT DISTINCT statement with RDS - mysql

Question

I have created a 2TB MySQL RDS, and filled it with 2 tables totaling 1.5TB:
+----------+---------------------------+------------+
| Database | Table | Size in MB |
+----------+---------------------------+------------+
| stam_db | owl | 1182043.00 |
| stam_db | owl_owners | 393695.00 |
The instance was set with db.m6g.2xlarge size and 6000 provisioned IOPS.
I ran this query to return the first 10 rows (they are all distinct, no duplicated rows):
SELECT DISTINCT *
FROM owl
ORDER BY
name
LIMIT 10;
To my surprise, this query has been running for the last 2 hours...
Even more surprising, the "Free Storage Space" AWS metric started to decrease at a rate of 2.2GB/minute:
For some reason, Write IOPS suddenly risen to 600-700 per second:
READ IOPS went even higher, to about 1850 per second:
This brings total IOPS to around 2400-2500:
CPU Utilization remained in the low single digits:
I have a few questions:
Why would a SELECT DISTINCT statement cause such massive writes into the database?
Why would the SELECT DISTINCT try to read the entire DB, instead of just the first 10 rows?
Why isn't RDS using the 6000 allocated IOPS? The total IOPS are only about 40% of the allocated amount.
For future reference, here are the answers:
Q2) I think I found an explanation at https://www.percona.com/blog/2019/07/17/mysql-disk-space-exhaustion-for-implicit-temporary-tables/ -" The queries that require a sorting stage most of the time need to rely on a temporary table. For example, when you use GROUP BY, ORDER BY or DISTINCT. Such queries are executed in two stages: the first is to gather the data and put them into a temporary table, the second is to execute the sorting on the temporary table." So even regular SELECT with ORDER BY needs to re-read then whole table
Q1) The massive writes are caused by the temporary table created for the query, they can reach 100% of the original table.
Q3) Looks like MySQL code creating the temporary tables simply isn't efficient enough to utilize the entire 6000 IOPS

Answer 1

Try to use EXPLAIN to analyze your SELECT DISTINCT query. I bet it will include "Using temporary" and/or "Using filesort". With a large enough result set, these queries will use temporary disk space. But the more frequently you run these queries, the more disk space it uses.
I don't know why you use SELECT DISTINCT * if the rows are already distinct. This may cause the use of a temporary table unnecessarily.
Ideally your query should be:
SELECT *
FROM owl
ORDER BY
name
LIMIT 10;
Make sure there is an index on the name column, so it can skip the "Using filesort" by reading rows in the index order by name.
Why isn't it using the full provisioned IOPS? I would guess because MySQL is constrained by the code that builds temporary tables. It can't fill the temp tables fast enough to saturate a high number of IOPS. Perhaps if you were to run this query concurrently in many threads it would. But maybe not. IMO, provisioned IOPS are pretty much a scam.