MySQL 5.6 long WHERE IN query very slow - mysql

Question

Since version 5.6 of MySQL a very simple albeit long query takes several orders longer than in 5.4.
The schema: Three tables, one with elements, one with categories and an M:N table tween those. Create Statements:
CREATE TABLE element (
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB AUTO_INCREMENT=4257455 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE category (
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB AUTO_INCREMENT=76 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE elements_categories (
id int(11) NOT NULL AUTO_INCREMENT,
element_id int(11) NOT NULL,
category_id int(11) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY element_id (element_id,category_id),
KEY elements_categories_element_id (element_id),
KEY elements_categories_category_id (category_id),
CONSTRAINT D7d489b06a407a0c1c70f108712c815e FOREIGN KEY (category_id) REFERENCES category (id),
CONSTRAINT co_element_id_57f4f2ec0db9441c_fk_element_id FOREIGN KEY (element_id) REFERENCES element (id)
) ENGINE=InnoDB AUTO_INCREMENT=88131737 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
The query:
SELECT elements_categories.element_id, category.id, category.name
FROM category
INNER JOIN elements_categories
ON category.id = elements_categories.category_id
WHERE elements_categories.element_id IN (1, 2, 3, ...)
So, the element table does not even play a role in this query, I already got a bunch of IDs from with with a previous query. (Disclaimer: I'm using an ORM and also inlining the first query did not make things faster.) The number of values in the IN clause can become very big, in my example 14240. That's not a problem, takes a tenth of a second or so. That's the execution plan:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+--------+---------------------------------------------------------------------------+------------+---------+---------------------------------+-------+--------------------------+
| 1 | SIMPLE | elements_categories | range | element_id,elements_categories_element_id,elements_categories.category_id | element_id | 4 | NULL | 42720 | Using where; Using index |
| 1 | SIMPLE | category | eq_ref | PRIMARY | PRIMARY | 4 | elements_categories.category_id | 1 | NULL |
When I add one more element, the execution time explodes to 60 seconds plus a fetch time of 200 seconds. The execution plan also changes to this:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------------+------+---------------------------------------------------------------------------+---------------------------------+---------+-------------+------+-------------+
| 1 | SIMPLE | category | ALL | PRIMARY | NULL | NULL | NULL | 75 | NULL |
| 1 | SIMPLE | elements_categories | ref | element_id,elements_categories_element_id,elements_categories_category_id | elements_categories_category_id | 4 | category.id | 760 | Using where |
range and eq_ref lookups exchanged for ALL and ref, order of tables switched, not using elements_categories.category_id as ref although it is the foreign key between those two tables. I don't get why the plan gets changed like this.
There are 75 categories and 4,300,000 elements and 1,600,000 assignments.
My guess is that I'm exceeding some size limit here, but cannot figure out which one. Also I didn't change anything from the MySQL 5.5 installation which stuck to the former execution plan all the time.

Answer 1

There are several ways to trick the optimizer to use the correct plan:
Add an index hint: ... JOIN elements_categories FORCE INDEX (element_id)...
Swap the tables around and make category a LEFT JOIN (assuming every elements_categories has a category). This is not a generic solution, but should work in this case.
Make a temp table with the element_id's and JOIN it in all of your queries instead of using IN (1,2,3...). You should also be able to use IN (SELECT id FROM <temp table>) instead of literals.

Answer 2

The reason that the optimizer chooses another plan when you have different parameters is that it looks at statistics from the tables and guess which index will remove the most rows, but this is a guess and can often be wrong.
If you know better you need to tell the optimizer what to do with an index hint like the first example #Vatev gives.
An interesting thing about the optimizer is that since an index adds an extra layer of indirection and thus potentially more reads it has to remove more than half the table to be considered useful by the optimizer. (I don't remember how much more than half...)
Another interesting feature of the optimizer is that if the index contains all information needed from a table it can avoid looking up the actual row so depending on your situation you might benefit from adding an extra column to the index. This optimization is used in the first query-plan "using index", but not the second. Thus adding "element_id" to your index "elements_categories_category_id" might speed things up. see http://dev.mysql.com/doc/refman/5.6/en/explain-output.html