Index absent from `possible_keys`… but only on some environments - mysql

Question

I have a software stack which I run myself, but also deploy to customer premises.
There is a particular query which runs very well in my environment, but runs terribly in the customer's environment.
I have confirmed using EXPLAIN that my environment's query planner sees that there is a great index available (and uses it). Whereas the same query in the customer's environment does not offer that index under possible_keys.
Here's the full query, somewhat anonymized:
SELECT t0.*,
t1.*,
t2.*,
t3.value
FROM table0 t0
LEFT OUTER JOIN table1 t1
ON t0.id = t1.table0_id
LEFT OUTER JOIN table2 t2
ON t1.id = t2.table1_id
AND t2.deleted = 0
LEFT OUTER JOIN table3 t3
ON t0.id = t3.table0_id
AND t3.type = 'whatever'
WHERE t0.business IN ('workcorp')
AND '2016-11-01 00:00:00' <= t0.date
AND t0.date < '2016-12-01 00:00:00'
ORDER BY t0.date DESC
The stage where our environments differ is on JOINing to table3. So theoretically you can ignore a large amount of the query and think of it like this:
SELECT t0.*
t3.value
FROM table0 t0
LEFT OUTER JOIN table3 t3
ON t0.id = t3.table0_id
AND t3.type = 'whatever'
Both of our environments' query plans agree on how to JOIN to t1 and to t2. But they differ in their plan for how to JOIN to t3.
My environment correctly identifies two possible indexes for JOINing to t3, and correctly identifies that table0_id is the best choice for this query:
+----+-------------+-------+------+--------------------------+-----------+---------+------+-------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------+--------------------------+-----------+---------+------+-------+----------+-------------+
| 1 | SIMPLE | t3 | ref | table0_id,type_and_value | table0_id | 108 | func | 2 | 100.00 | Using where |
+----+-------------+-------+------+--------------------------+-----------+---------+------+-------+----------+-------------+
The customer's environment does not consider the index table0_id to be an option, and falls back to type_and_value (which is a really bad choice):
+----+-------------+-------+------+----------------+----------------+---------+-------+----------------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------+----------------+----------------+---------+-------+----------------+----------+-------------+
| 1 | SIMPLE | t3 | ref | type_and_value | type_and_value | 257 | const | (far too many) | 100.00 | Using where |
+----+-------------+-------+------+----------------+----------------+---------+-------+----------------+----------+-------------+
What happens if we FORCE INDEX?
EXPLAIN EXTENDED SELECT t0.*,
t1.*,
t2.*,
t3.value
FROM table0 t0
LEFT OUTER JOIN table1 t1
ON t0.id = t1.table0_id
LEFT OUTER JOIN table2 t2
ON t1.id = t2.table1_id
AND t2.deleted = 0
LEFT OUTER JOIN table3 t3 FORCE INDEX (table0_id)
ON t0.id = t3.table0_id
AND t3.type = 'whatever'
WHERE t0.business IN ('workcorp')
AND '2016-11-01 00:00:00' <= t0.date
AND t0.date < '2016-12-01 00:00:00'
ORDER BY t0.date DESC
On my environment, I got:
+----+-------------+-------+------+---------------+-----------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+-----------+---------+------+------+-------------+
| 1 | SIMPLE | t3 | ref | table0_id | table0_id | 108 | func | 2 | Using where |
+----+-------------+-------+------+---------------+-----------+---------+------+------+-------------+
Compared to my original query plan (which proposed two possible_keys): this narrowed down the choice to just one.
But the customer got a different result:
+----+-------------+-------+------+---------------+------+---------+-------+---------+----------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------+---------------+------+---------+-------+---------+----------+----------------------------------------------------+
| 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | (loads) | 100.00 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+-------+------+---------------+------+---------+-------+---------+----------+----------------------------------------------------+
Adding the FORCE INDEX narrows down the customer's possible_keys from one bad choice, to no choices.
So why is it that the customer's environment does not have the same indexes available in possible_keys? Naturally I was given to suspect "maybe they don't have that index". So we did a SHOW INDEXES FROM table3. Here's my environment (for comparison):
+--------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| table3 | 0 | PRIMARY | 1 | id | A | 16696 | NULL | NULL | | BTREE | | |
| table3 | 1 | table0_id | 1 | table0_id | A | 16696 | NULL | NULL | | BTREE | | |
| table3 | 1 | type_and_value | 1 | type | A | 14 | NULL | NULL | | BTREE | | |
| table3 | 1 | type_and_value | 2 | value | A | 8348 | NULL | NULL | | BTREE | | |
+--------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Their environment had the same index, table0_id available:
+--------+------------+-----------------+--------------+-----------------+-----------+-------------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------+------------+-----------------+--------------+-----------------+-----------+-------------------+----------+--------+------+------------+---------+---------------+
| table3 | 1 | table0_id | 1 | table0_id | A | (same as PRIMARY) | NULL | NULL | | BTREE | | |
+--------+------------+-----------------+--------------+-----------------+-----------+-------------------+----------+--------+------+------------+---------+---------------+
I was also careful to ask "is this a slave? is the master the same?": they assured me that all instances had this index, as required.
So I thought "maybe the index is broken in some way?" And proposed that they do the very simplest query relying upon that index:
EXPLAIN EXTENDED SELECT *
FROM table3
WHERE table0_id = 'whatever'
In this case: their environment behaved the same as mine (and correctly), proposing the use of the index table0_id:
+----+-------------+--------+------+---------------+-----------+---------+-------+------+----------+-----------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------+---------------+-----------+---------+-------+------+----------+-----------------------+
| 1 | SIMPLE | table3 | ref | table0_id | table0_id | 108 | const | 1 | 100.00 | Using index condition |
+----+-------------+--------+------+---------------+-----------+---------+-------+------+----------+-----------------------+
So they definitely have that index. And their query planner can recognise that it is eligible for use (for some queries at least).
What's going on here? Why is table0_id ineligible for use on certain queries, but only on the customer's environment? Could it be that the index is broken in some way? Or that the query planner is making a mistake?
Are there any other tests I can do to figure out why it's not using the index for this query?

Answer 1

Turns out it was the charsets (and/or the collations).
I used this query to reveal the charset in each environment:
SELECT table_name, column_name, character_set_name FROM information_schema.`COLUMNS`
WHERE table_schema = "my_cool_database"
AND table_name IN ("t0", "t3")
ORDER BY 1 DESC, 2
And for bonus points I checked the character collation in each environment:
SHOW FULL COLUMNS FROM t0;
SHOW FULL COLUMNS FROM t3;
In my environment: all columns in both tables had utf8 charset and utf8_unicode_ci collation.
In the customer's environment: t0 matched my environment exactly, yet t3 was a unique snowflake; its columns had latin1 charset and latin1_swedish_ci collation.
So, what we were seeing is that the index that existed on t3.table0_id (a latin1 column) could not be used to JOIN to a utf8 table. Hence the index worked fine for:
SELECT *
FROM table3
WHERE table0_id = 'whatever'
Yet the index could not be used for:
SELECT
t0.id,
t3.value
FROM t0
LEFT OUTER JOIN t3
Similar symptoms are described on the Percona blog, John Watson's blog and Baron Schwartz's blog.