Optimizing query in MySQL5.6 - mysql

Question

I have an INNODB table levels:
+--------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| id | int(9) | NO | PRI | NULL | |
| level_name | varchar(20) | NO | | NULL | |
| user_id | int(10) | NO | | NULL | |
| user_name | varchar(45) | NO | | NULL | |
| rating | decimal(5,4) | NO | | 0.0000 | |
| votes | int(5) | NO | | 0 | |
| plays | int(5) | NO | | 0 | |
| date_published | date | NO | MUL | NULL | |
| user_comment | varchar(255) | NO | | NULL | |
| playable_character | int(2) | NO | | 1 | |
| is_featured | tinyint(1) | NO | MUL | 0 | |
+--------------------+--------------+------+-----+---------+-------+
There are ~4 million rows. Because of the front-end functionality, I need to query this table with a variety of filters and sorts. They are on playable_character, rating, plays, and date_published. The date_published can be filtered to show by the last day, week, month, or anytime(last 3 years). There's also paging. So, depending on the user choices, the queries can look, for example, like one of these:
SELECT * FROM levels
WHERE playable_character = 0 AND
date_published BETWEEN date_sub(now(), INTERVAL 3 YEAR) AND now()
ORDER BY date_published DESC
LIMIT 0, 1000;
SELECT * FROM levels
WHERE playable_character = 4 AND
date_published BETWEEN date_sub(now(), INTERVAL 1 WEEK) AND now()
ORDER BY rating DESC
LIMIT 4000, 1000;
SELECT * FROM levels
WHERE playable_character = 5 AND
date_published BETWEEN date_sub(now(), INTERVAL 1 MONTH) AND now()
ORDER BY plays DESC
LIMIT 1000, 1000;
I started out with an index idx_date_char(date_published, playable_character) that works great on the first example query here -- basically anything that's ordering by date_published. Using EXPLAIN, I get 'using index condition', which is good. I think I understand why the index works, since the same two indexed columns exist in the WHERE and ORDER BY clauses.
My problem is with queries that ORDER by plays or rating. I understand I'm introducing a third column, but for the life of me I can't get an index that works well, despite trying just about every variation I could think of: composite indexes of all three or four in every order, and so on. Maybe the query could be written differently?
I should add that rating and plays are always queried as DESC. Only date_published may be either DESC or ASC.
Any suggestions greatly appreciated. TIA.

Answer 1

It seems you would make good use of data sorted in this way for each of the queries:
playable_character, date_published
playable_character, date_published, rating
playable_character, date_published, plays
Bear in mind that the data you need sorted in the first query happens to be a subset of the data the second and third query needs, so we can get rid of it.
Also note that adding DESC or ASC to an index is syntactically correct but doesn't actually change anything as that feature is not currently supported (it is expected to be supported in the future so that is why it is there). All indexes are stored in ascending order. More information here.
So these are the indexes that you should create:
ALTER TABLE levels ADD INDEX (playable_character, date_published, rating)
ALTER TABLE levels ADD INDEX (playable_character, date_published, plays)
That should make the 3 queries up there run faster than Forrest Gump.

Answer 2

The columns used in your where clause AND order by should be part of the index. I would have an index on
( playable_character, date_published DESC, rating DESC, plays DESC )
The reason I would put the playable character FIRST is you want that ID primary, then all those dates within question. The rating and plays are just along for the ride for assisting the ORDER BY clause).
Think of the index like this. If you have it ordered by Date_Published, then Playable_Character, think of a room of boxes. Each box has a date.. Within that box for a given date, you have them in order of character. So, you have 3 years worth of data to go through, you have to open all boxes for the last 3 years and find the character you are looking for.
Now, think of it like this. Each box is by character, and within that, all their dates are pre-sorted. So, you go to one box, open it... Move to the date in question and grab the records from X-Y range you want. Now, you can apply a simple order by of those records.

Answer 3

When your query includes a range predicate like BETWEEN, the order of columns in your index is important.
First, include one or more columns referenced by equality predicates.
Next, include one column referenced by a range predicate.
Any further columns in the index after the column referenced by a range predicate cannot be used for other range predicates or for sorting.
If you have no range predicate, you can add a column for sort order.
So your first query can benefit from an index on (playable_character, date_published). The sorting should be a no-op because the optimizer will just fetch rows in the index order.
The second and third queries are bound to do a filesort, because you have a range predicate and then you're sorting by a different column. If you had had only equality predicates, you would be able to use the third column to avoid the filesort, but that doesn't work when you have a range predicate.
The best you can hope for is that the conditions reduce the size of the result set so that it can sort in memory without doing too many sort merge passes. You can help this by increasing sort_buffer_size, but be careful not to increase it too much, because it's allocated per thread.
The ASC/DESC keywords in index definitions makes no difference in MySQL.
See http://dev.mysql.com/doc/refman/5.6/en/create-index.html:
These keywords are permitted for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.