Why does DISTINCT make this query take 10x longer than without? - mysql

Question

I've got this mysql query:
SELECT DISTINCT post.postId,hash,previewUrl,lastRetrieved
FROM post INNER JOIN (tag as t1,taggedBy as tb1,tag as t2,taggedBy as tb2,tag as t3,taggedBy as tb3)
ON post.id=tb1.postId AND tb1.tagId=t1.id AND post.id=tb2.postId AND tb2.tagId=t2.id AND post.id=tb3.postId AND tb3.tagId=t3.id
WHERE ((t1.name="a" AND t2.name="b") OR t3.name="c")
ORDER BY post.postId DESC LIMIT 0,100;
it takes around 15 seconds to run that query, whereas the same query without DISTINCT takes less than a second.
EXPLAIN output for the query with DISTINCT:
+----+-------------+-------+--------+---------------------+---------+---------+--------------------------+------+-----------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------------+---------+---------+--------------------------+------+-----------------------+
| 1 | SIMPLE | post | index | PRIMARY | postId | 4 | NULL | 1 | Using temporary |
| 1 | SIMPLE | tb1 | ref | PRIMARY,tagId | PRIMARY | 4 | e621datamirror.post.id | 13 | Using index; Distinct |
| 1 | SIMPLE | t1 | eq_ref | PRIMARY,name,name_2 | PRIMARY | 4 | e621datamirror.tb1.tagId | 1 | Distinct |
| 1 | SIMPLE | tb2 | ref | PRIMARY,tagId | PRIMARY | 4 | e621datamirror.post.id | 13 | Using index; Distinct |
| 1 | SIMPLE | t2 | eq_ref | PRIMARY,name,name_2 | PRIMARY | 4 | e621datamirror.tb2.tagId | 1 | Distinct |
| 1 | SIMPLE | tb3 | ref | PRIMARY,tagId | PRIMARY | 4 | e621datamirror.post.id | 13 | Using index; Distinct |
| 1 | SIMPLE | t3 | eq_ref | PRIMARY,name,name_2 | PRIMARY | 4 | e621datamirror.tb3.tagId | 1 | Using where; Distinct |
+----+-------------+-------+--------+---------------------+---------+---------+--------------------------+------+-----------------------+
7 rows in set (0.01 sec)
EXPLAIN output for the query without DISTINCT:
+----+-------------+-------+--------+---------------------+---------+---------+--------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------------+---------+---------+--------------------------+------+-------------+
| 1 | SIMPLE | post | index | PRIMARY | postId | 4 | NULL | 1 | NULL |
| 1 | SIMPLE | tb1 | ref | PRIMARY,tagId | PRIMARY | 4 | e621datamirror.post.id | 13 | Using index |
| 1 | SIMPLE | t1 | eq_ref | PRIMARY,name,name_2 | PRIMARY | 4 | e621datamirror.tb1.tagId | 1 | NULL |
| 1 | SIMPLE | tb2 | ref | PRIMARY,tagId | PRIMARY | 4 | e621datamirror.post.id | 13 | Using index |
| 1 | SIMPLE | t2 | eq_ref | PRIMARY,name,name_2 | PRIMARY | 4 | e621datamirror.tb2.tagId | 1 | NULL |
| 1 | SIMPLE | tb3 | ref | PRIMARY,tagId | PRIMARY | 4 | e621datamirror.post.id | 13 | Using index |
| 1 | SIMPLE | t3 | eq_ref | PRIMARY,name,name_2 | PRIMARY | 4 | e621datamirror.tb3.tagId | 1 | Using where |
+----+-------------+-------+--------+---------------------+---------+---------+--------------------------+------+-------------+
CREATE TABLE `post` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`postId` int(11) NOT NULL,
`hash` varchar(32) COLLATE utf8_bin NOT NULL,
`previewUrl` varchar(512) COLLATE utf8_bin NOT NULL,
`lastRetrieved` datetime NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `postId` (`postId`),
UNIQUE KEY `hash` (`hash`),
KEY `postId_2` (`postId`),
KEY `postId_3` (`postId`)
) ENGINE=InnoDB AUTO_INCREMENT=692561 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `tag` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`),
KEY `name_2` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=157876 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `taggedBy` (
`postId` int(11) NOT NULL,
`tagId` int(11) NOT NULL,
PRIMARY KEY (`postId`,`tagId`),
KEY `tagId` (`tagId`),
CONSTRAINT `taggedBy_ibfk_1` FOREIGN KEY (`postId`) REFERENCES `post` (`id`) ON DELETE CASCADE,
CONSTRAINT `taggedBy_ibfk_2` FOREIGN KEY (`tagId`) REFERENCES `tag` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
what causes this query to be so slow? how can I speed it up?
I hope I've given enough information so you guys can give me some meaningful answers. if I've left something out I'll be happy to add it.

Answer 1

Several things are being discussed, even in #SlimGhost's reasonable (but deleted) answer.
DISTINCT vs GROUP BY
Although GROUP BY can sometimes be used to replace DISTINCT, don't do it; they are meant for different things.
They both require some form of extra effort. (I'll get to the 10x later.) Both have to discover common values -- either in the entire row (for DISTINCT) or for the grouped items. This can be done in one of at least two ways. (Probably most engines have these options built in.) Note that the DISTINCT or GROUP BY must logically come after WHERE, but before ORDER BY and LIMIT.
Keep some kind of internal associative array as the output is being generated. This is practical if the the optimizer can see that there won't be "too many" possible different values.
Sort the output; then dedup or group in a pass over the output. This works regardless of size.
ORDER BY + LIMIT
Notice that the query is doing DISTINCT over 4 columns: post.postId, hash, previewUrl, lastRetrieved. It is not obvious whether these are all in post or scattered across the 7 tables. (Please clarify by qualifying every column.)
Let's assume the JOINs need to be done to find the 4 columns.
Let's say there is no DISTINCT. Now, the operations are
Walk through post in ORDER BY post.postID order.
For each such row, do the JOINs and check the WHERE.
After 100 rows have passed the WHERE, stop.
But with DISTINCT, the optimizer can't make such a simplifying assumption in order to stop short. Instead:
Walk through post in ORDER BY post.postID order. (Starting with t1/t2/t3 is out of the question because of OR.) Actually, it is unclear whether the optimizer would bother going in this order.
For each such row, do the JOINs and check the WHERE.
Do something about DISTINCT.
After 100 rows have passed the WHERE, stop. Note: This may involve lots more rows from post (perhaps 10x?)
Keep in mind that the optimizer knows nothing about whether postId is 1:1 with hash, etc. So, it can't make simplifying assumptions. Suppose there were 200 rows in the JOIN with the smallest postId, and the hash happened to be in descending order. Smells like a need for a "sort".
EXPLAIN FORMAT=JSON SELECT ... might give you some of these details.
Ouch. You have both a id and UNIQUE(postid)? Get rid of id and turn the postId into the PRIMARY KEY. This, alone, may speed things up.
What is the hash a hash of?
Please use the JOIN ... ON ... syntax.
You have 3 indexes on postId; get rid of the extra two.
Why use DISTINCT?
Now that I see that all the SELECTed columns come from the one table, and that they will obviously be easily made distinct, why even consider using DISTINCT.
(updates)
JOIN ON
FROM post INNER JOIN (tag as t1,taggedBy as tb1,...
ON post.id=tb1.postId AND tb1.tagId=t1.id AND ...
-->
FROM post
JOIN tag AS t1 ON post.id = tb1.postId
JOIN taggedBy AS tb1 ON tb2.tagId = t2.id
... (each ON is next to the JOIN it applies to)
A speedup technique
SELECT p2.postId, p2.hash, p2.previewUrl, p2.lastRetrieved
FROM (
SELECT DISTINCT postId -- Only the PRIMARY KEY
FROM post
JOIN ... etc
WHERE ... ...
ORDER BY postId
LIMIT 100
) x
JOIN post AS p2 ON x.postId = p2.id -- self join for getting rest of fields
ORDER BY x.postId -- assuming you need the ordering
This puts the DISTINCT in the inner query, where you are fetching only the one column (postId). (I am not sure whether this technique will help much in your case.)