MySQL doesn't use indexes in JOIN query - mysql

Question

I have a main table with 500000+ rows.
CREATE TABLE `esc_questions`(
`id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`esc_id` INT(11) NOT NULL,
`question_text` LONGTEXT COLLATE utf8_unicode_ci NOT NULL,
`answer_1` TEXT COLLATE utf8_unicode_ci NOT NULL,
`answer_2` TEXT COLLATE utf8_unicode_ci NOT NULL,
`answer_3` TEXT COLLATE utf8_unicode_ci NOT NULL,
`answer_4` TEXT COLLATE utf8_unicode_ci NOT NULL,
`answer_5` TEXT COLLATE utf8_unicode_ci NOT NULL,
`right_answer` VARCHAR(255) COLLATE utf8_unicode_ci NOT NULL,
`disciplinas_id` INT(11) UNSIGNED NOT NULL,
`assunto_id` INT(11) UNSIGNED NOT NULL,
`orgao_id` INT(11) UNSIGNED NOT NULL,
`cargo_id` INT(11) UNSIGNED NOT NULL,
`ano` INT(11) NOT NULL,
`banca_id` INT(11) UNSIGNED NOT NULL,
`question_type` TINYINT(4) NOT NULL,
`url` TEXT COLLATE utf8_unicode_ci NOT NULL,
`created_at` TIMESTAMP NULL DEFAULT NULL,
`updated_at` TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY(`id`),
KEY `idx_ano`(`ano`) USING BTREE,
KEY `idx_question_type`(`question_type`) USING BTREE,
KEY `idx_cargo_id`(`cargo_id`) USING BTREE,
KEY `idx_orgao_id`(`orgao_id`) USING BTREE,
KEY `idx_banca_id`(`banca_id`) USING BTREE,
KEY `idx_question_id`(`id`) USING BTREE,
KEY `idx_assunto_id`(`assunto_id`) USING BTREE,
KEY `idx_disciplinas_id`(`disciplinas_id`) USING BTREE,
CONSTRAINT `fk_assunto_id` FOREIGN KEY(`assunto_id`) REFERENCES `esc_assunto`(`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_banca_id` FOREIGN KEY(`banca_id`) REFERENCES `esc_bancas`(`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_cargo_id` FOREIGN KEY(`cargo_id`) REFERENCES `esc_cargo`(`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_disciplinas_id` FOREIGN KEY(`disciplinas_id`) REFERENCES `esc_disciplinas`(`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_orgao_id` FOREIGN KEY(`orgao_id`) REFERENCES `esc_orgao`(`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE = INNODB AUTO_INCREMENT = 516157 DEFAULT CHARSET = utf8 COLLATE = utf8_unicode_ci
Related data is stored to five additional tables, very similar to this one:
CREATE TABLE `esc_assunto`(
`id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(255) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY(`id`),
KEY `idx_assunto_id`(`id`) USING BTREE,
KEY `idx_assunto_name`(`name`(30)),
CONSTRAINT `fk_assunto` FOREIGN KEY(`id`) REFERENCES `esc_questions`(`assunto_id`) ON DELETE NO ACTION ON UPDATE NO ACTION) ENGINE = INNODB AUTO_INCREMENT = 3618 DEFAULT CHARSET = utf8 COLLATE = utf8_unicode_ci
I have pagination on my website. When I'm trying to get latest pages, time taken for data request is rising.
Here is my SELECT for this task:
SELECT
f.*,
d.name disciplinas,
o.name orgao,
c.name cargo,
b.name banca,
a.name assunto
FROM
`esc_questions` f
INNER JOIN
`esc_bancas` b
ON
f.banca_id = b.id
INNER JOIN
`esc_disciplinas` d
ON
f.disciplinas_id = d.id
INNER JOIN
`esc_assunto` a
ON
f.assunto_id = a.id
INNER JOIN
`esc_orgao` o
ON
f.orgao_id = o.id
INNER JOIN
`esc_cargo` c
ON
f.cargo_id = c.id
LIMIT 400020, 20
This query takes a long time on Sending Data stage showed in query profiler.
Sending Data 17.6 s 99.99% 1 17.6 s
EXPLAIN shows the following:
1 SIMPLE d ALL PRIMARY,idx_disciplinas_id 247
1 SIMPLE f ref idx_cargo_id,idx_orgao_id,idx_banca_id,idx_assunto_id,idx_disciplinas_id idx_disciplinas_id 4 concursos.d.id 1116
1 SIMPLE o eq_ref PRIMARY,idx_orgao_id PRIMARY 4 concursos.f.orgao_id 1
1 SIMPLE c eq_ref PRIMARY,idx_cargo_id PRIMARY 4 concursos.f.cargo_id 1
1 SIMPLE a eq_ref PRIMARY,idx_assunto_id PRIMARY 4 concursos.f.assunto_id 1
1 SIMPLE b eq_ref PRIMARY,idx_bancas_id PRIMARY 4 concursos.f.banca_id 1
I spent all day to make this work fast, but no success.
Can somebody tell me what's wrong with my select query or why MySQL doesn't use indexes?
Any help appreciated.

Answer 1

You have the wrong approach in several says. First, your query has no order by clause. A query is not guaranteed to return the results in the same order on multiple executions (although in practice queries do, debugging such a problem could be really hard).
So, you should add an order by, probably on the primary key of esc_questions and whatever secondary keys are necessary.
Second, the offset of 400020 is rather large. MySQL is going to generate 400,020 rows and discard them, before finding the 400,021st row.
My suggestion is to find the "id" used in the sort and then include a where clause:
where ?? > $last_id
. . .
order by ??
limit 20
This may not (or may) speed up the load the first time, but it should speed subsequent loads.

Answer 2

I found solution myself. I need to avoid LIMIT with offset in my JOIN query.
In order to do this I need to do some preparation:1. Get only ids from my main table without any joins on needed offset. This query took 0.0856 seconds
SELECT id FROM `esc_questions` WHERE 1 LIMIT 489980, 20
2. Create composite index in order you will then make your query. In my case I use the following index:
...
KEY `idx_filter_search` (`id`,`disciplinas_id`,`assunto_id`,`orgao_id`,`cargo_id`,`banca_id`) USING BTREE,
...
3. Finally make your query. Query took 0.0040 seconds:
SELECT SQL_NO_CACHE
f.*,
d.name disciplinas,
o.name orgao,
c.name cargo,
b.name banca,
a.name assunto
FROM
`esc_questions` f FORCE INDEX(idx_filter_search),
`esc_disciplinas` d,
`esc_assunto` a,
`esc_orgao` o,
`esc_cargo` c,
`esc_bancas` b
WHERE
f.id IN(
497442,
497444,
497445,
497447,
497449,
497450,
497452,
497453,
497454,
497456,
497458,
497459,
497461,
497462,
497464,
497465,
497467,
497468,
497470,
497471
) AND f.disciplinas_id = d.id AND f.assunto_id = a.id AND f.orgao_id = o.id AND f.cargo_id = c.id AND f.banca_id = b.id
ORDER BY
id
EXPLAIN this query will tell me that it's using my newly created index.
1 | SIMPLE | f | range | idx_filter_search | idx_filter_search | 4 | NULL | 20 | Using where
Hope this helps someone.
Thanks #GordonLinoff for pointing me to the right direction.