How to index 1 million rows mySQL table for simple query - mysql

Question

I have a mySQL DB of ~1 milllion entries.
I run the query:
SELECT a.id as aid, a.title as atitle, a.slug, summary,
a.link as alink, author, published, image, a.cat as acat,
a.rss as arss, a.site as asite
FROM articles a
ORDER BY published DESC
LIMIT 616150, 50;
It takes about 5 minutes or more ot load.
My TABLE and INDEXes:
CREATE TABLE IF NOT EXISTS `articles` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`slug` varchar(255) NOT NULL,
`summary` text NOT NULL,
`link` text NOT NULL,
`author` varchar(255) NOT NULL,
`published` datetime NOT NULL,
`image` text NOT NULL,
`cat` int(11) NOT NULL,
`rss` int(11) NOT NULL,
`site` int(11) NOT NULL,
`bitly` varchar(255) NOT NULL,
`checked` tinyint(4) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `title` (`title`),
KEY `cat` (`cat`),
KEY `published` (`published`),
KEY `site` (`site`),
KEY `rss` (`rss`),
KEY `checked` (`checked`),
KEY `id_publ_index` (`id`,`published`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1230234;
What explain says:
mysql> EXPLAIN EXTENDED SELECT a.id as aid, a.title as atitle, a.slug, summary, a.link as alink, author, published, image, a.cat as acat, a.rss as arss, a.site as asite FROM articles a ORDER BY published DESC LIMIT 616150, 50;
+----+-------------+-------+-------+---------------+-----------+---------+------+--------+----------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+-------+---------------+-----------+---------+------+--------+----------+-------+
| 1 | SIMPLE | a | index | NULL | published | 8 | NULL | 616200 | 152.94 | |
+----+-------------+-------+-------+---------------+-----------+---------+------+--------+----------+-------+
1 row in set, 1 warning (0.46 sec)
Any tips of how to optimize this query? Why mySQL needs to read all 616200 rows and not just the 50 that were asked?
Thank you for your time.

Answer 1

The reason that you are seeing the published key being used is because that is what you are ordering by. How often does this query need to run?
There is one simple thing that you can do to help this query run much, much faster:
Make better use of your published key. Use WHERE to define what range of dates you want to retrieve from your table.
The reason why you are reading 616,200 rows of your table right now is because you are not using the index to limit the range. MySQL needs to use your full index to:
Sort the first 616200 rows in DESC order and then
Finally limit the result to 50 rows.
If possible, you should filter the results of your database in a different way. Changing your results to be based on the WHERE (making more efficient use of your index) will be the quickest way.
For example:
SELECT a.id as aid, a.title as atitle, a.slug, summary,
a.link as alink, author, published, image, a.cat as acat,
a.rss as arss, a.site as asite
FROM articles a
WHERE published > '2010-01-01'
ORDER BY published DESC
LIMIT 6150, 50;
The sad part is that ORDER BY and LIMIT does not scale too well, and you will loose you speed very quickly. (eg, change your limit to 0, 50, and then to 900000, 50 and see how your speed is affected) so adding more information to your WHERE will help your query be much faster.
EDIT:
There is no way I can know what to display by date, so putting a where is not possible. In addition this query is run on a news aggregator, that collects news every ... second. The limit is made so I can create paginated results.
Because you are inserting new posts, your LIMIT statement is going to cause the news items to jump when a user is going through page anyway. For instance, if I am on page one and three items get added before I press 'Next', then by the time that I click 'Next', I will see the last three items from the previous page.
For the best possible user experience, you should try adding the ID of the last seen news item or the date of the last seen news item to the pagination somehow. This could be done either by sessions or part of the query URL, but it will allow you to have better use of your indexes.
I understand why the limit is there - it's just how can you fix the issue of the query being slow after a certain amount of pages have been hit.
To efficiently fix your speed issues, you will need to make better use of an index, rather then relying on 'LIMIT' to be your sole method of pagination. LIMIT is amazing, yes, but it is not optimized for retrieving records the way that you are trying to do it, because you need to sort by the date.
Even though as you say 'there is no way I can know what to display by date' (at least currently...) there must be a way for your application to limit what needs to be fetched from your db. The same way Facebook does not need to go through every member of the website's individual posts just to make results show on your Facebook wall. You need to find out how it can be made more efficient.