Filesort query, has index, but not using it - mysql

Question

Why is the below query failing it use the index story_id, in the story_keywords table?
mysql> EXPLAIN SELECT `stories`.*
-> FROM (`stories`)
-> JOIN `story_keywords` ON `story_keywords`.`story_id` = `stories`.`id`
-> WHERE `image_full_url` != ''
-> AND `order` != 0
-> AND `news_type` IN ('movie', 'movie_review')
-> AND `keyword` IN ('topnews', 'toptablet')
-> GROUP BY `stories`.`id`
-> ORDER BY `created` DESC, `order` DESC
-> LIMIT 5 ;
+----+-------------+----------------+--------+---------------+---------+---------+---------------------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------+--------+---------------+---------+---------+---------------------------------------+------+----------------------------------------------+
| 1 | SIMPLE | story_keywords | ALL | story_id | NULL | NULL | NULL | 42 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | stories | eq_ref | PRIMARY | PRIMARY | 767 | entertainment.story_keywords.story_id | 1 | Using where |
+----+-------------+----------------+--------+---------------+---------+---------+---------------------------------------+------+----------------------------------------------+
2 rows in set (0.00 sec)
mysql> show create table stories;
+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| stories | CREATE TABLE `stories` (
`id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`news_type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`created` datetime DEFAULT NULL,
`author` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`author_title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`image_caption` text COLLATE utf8_unicode_ci,
`image_credit` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`image_full_url` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`body` text COLLATE utf8_unicode_ci,
`summary` text COLLATE utf8_unicode_ci,
`external_url` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`order` int(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> show create table story_keywords;
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| story_keywords | CREATE TABLE `story_keywords` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`story_id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`keyword` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `story_id` (`story_id`)
) ENGINE=MyISAM AUTO_INCREMENT=85 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Answer 1

It is probably because MySQL believes it is cheaper to fetch ALL rows from story_keywords table and JOIN them in instead of using indexes. It sounds weird at first, but, you see, if you have to perform 100 index lookups on a table and this table has just about 100 rows – it will cost less to read all rows. The explanation is simple: index lookup (for BTREE indexes) is O(ln N), while reading N rows is O(N). Obviously, O(N) < N * O(ln N).
To prove it – try selecting just 1 row from stories (and by one I mean one row, not sorting the whole table and limiting the result ;), just like:
SELECT `stories`.*
FROM (`stories`)
JOIN `story_keywords` ON `story_keywords`.`story_id` = `stories`.`id`
WHERE `stories`.id = SOMETHING
This query is much more likely to turn to index on story_keywords.
Hope this answers your question :)

Answer 2

Anton is on the right track, but I believe there is more to the problem. As my comment on the OP says, the id columns should most likely be INT types. As the explain shows, the length of the primary key on stories is 767. Usually for an INT type the length would be in the low single digits, but since the column is a VARCHAR, the length is extremely long.
Back to the main problem, since there are no indexes on stories.news_type, stories.order, or story_keywords.story_keywords, the optimizer decided to do a full scan of story_keywords since it will yield the smallest initial result set. If there was an index on one of those columns, it would likely use that first. If you add an index that the query can use it will not need to do a full table scan.