How to optimise this slow MySQL query - late row lookups? - mysql

Question

I'm converting a site over to use XenForo as forum software, however this site has millions of thread rows in the MySQL table. If I try to browse a paginated listing of threads, it slows to a crawl the further I go. Once I'm at page 10,000 it takes almost 30s.
My aim is to improve the query below, perhaps by using late row lookups so that I can make this query run faster:
SELECT thread.*
,
user.*, IF(user.username IS NULL, thread.username, user.username) AS username,
NULL AS thread_read_date,
0 AS thread_is_watched,
0 AS user_post_count
FROM xf_thread AS thread
LEFT JOIN xf_user AS user ON
(user.user_id = thread.user_id)
WHERE (thread.node_id = 152) AND (thread.sticky = 0) AND (thread.discussion_state IN ('visible'))
ORDER BY thread.last_post_date DESC
LIMIT 20 OFFSET 238340
Run Time: 4.383607
Select Type Table Type Possible Keys Key Key Len Ref Rows Extra
SIMPLE thread ref node_id_last_post_date,node_id_sticky_state_last_post node_id_last_post_date 4 const 552480 Using where
SIMPLE user eq_ref PRIMARY PRIMARY 4 sitename.thread.user_id 1
Schema:
CREATE TABLE `xf_thread` (
`thread_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`node_id` INT(10) UNSIGNED NOT NULL,
`title` VARCHAR(150) NOT NULL,
`reply_count` INT(10) UNSIGNED NOT NULL DEFAULT '0',
`view_count` INT(10) UNSIGNED NOT NULL DEFAULT '0',
`user_id` INT(10) UNSIGNED NOT NULL,
`username` VARCHAR(50) NOT NULL,
`post_date` INT(10) UNSIGNED NOT NULL,
`sticky` TINYINT(3) UNSIGNED NOT NULL DEFAULT '0',
`discussion_state` ENUM('visible','moderated','deleted') NOT NULL DEFAULT 'visible',
`discussion_open` TINYINT(3) UNSIGNED NOT NULL DEFAULT '1',
`discussion_type` VARCHAR(25) NOT NULL DEFAULT '',
`first_post_id` INT(10) UNSIGNED NOT NULL,
`first_post_likes` INT(10) UNSIGNED NOT NULL DEFAULT '0',
`last_post_date` INT(10) UNSIGNED NOT NULL,
`last_post_id` INT(10) UNSIGNED NOT NULL,
`last_post_user_id` INT(10) UNSIGNED NOT NULL,
`last_post_username` VARCHAR(50) NOT NULL,
`prefix_id` INT(10) UNSIGNED NOT NULL DEFAULT '0',
`sonnb_xengallery_import` TINYINT(3) DEFAULT '0',
PRIMARY KEY (`thread_id`),
KEY `node_id_last_post_date` (`node_id`,`last_post_date`),
KEY `node_id_sticky_state_last_post` (`node_id`,`sticky`,`discussion_state`,`last_post_date`),
KEY `last_post_date` (`last_post_date`),
KEY `post_date` (`post_date`),
KEY `user_id` (`user_id`)
) ENGINE=INNODB AUTO_INCREMENT=2977 DEFAULT CHARSET=utf8
Can anyone help me improve the speed of this query? I'm a real MySQL novice, but I am running the same dataset on other forum software and it is much faster - so I'm sure there is a way somehow. This table is INNODB and I'd consider the server well optimised.

Answer 1

This might help: http://explainextended.com/2009/10/23/mysql-order-by-limit-performance-late-row-lookups/
The concept being, query just the index column with your required paging/ordering, then join this list to the other columns you want from the table

Answer 2

Your User table is already index by user ID... good.
For your thread table, I would have a compound index on it with the key
( note_id, sticky, discussion_state, last_post_date )
This way, the index is optimized on all parts in the WHERE clause... AND since it has the last_post_date too, that can be utilized by the ORDER BY clause. Order By clauses are notorious for killing query performance.