MySQL multiple index condition become slower? - mysql

Question

I have a table defined like this:
article | CREATE TABLE `article` (
`id` varchar(64) NOT NULL,
`type` varchar(16) DEFAULT NULL,
`title` varchar(1024) DEFAULT NULL,
`source` varchar(64) DEFAULT NULL,
`over` tinyint(1) DEFAULT NULL,
`taken` tinyint(1) DEFAULT NULL,
`released_at` varchar(32) DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_article_over` (`over`),
KEY `idx_article_created_at` (`created_at`),
KEY `idx_article_type` (`type`),
KEY `idx_article_taken` (`taken`),
KEY `idx_article_updated_at` (`updated_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
mysql> select count(1) from article;
+----------+
| count(1) |
+----------+
| 649773 |
+----------+
1 row in set (0.61 sec)
when I make a query:
SELECT * FROM `article` where taken=0 ORDER BY updated_at asc limit 10;
or
SELECT * FROM `article` where over=0 ORDER BY updated_at asc limit 10;
They are both very fast.
However when I use this, it become very slow:
SELECT * FROM `article` where taken=0 and over=0 ORDER BY updated_at asc limit 10;
It takes 4.94s.
If the article table grows to 20 million rows, it takes much longer time.
Here is the explain with 20 million rows:
mysql> explain SELECT * FROM `article` where taken=0 and processed=0 ORDER BY updated_at asc limit 10;
+----+-------------+-----------+------------+-------------+---------------------------------------------+---------------------------------------------+---------+------+---------+----------+-------------------------------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------+------------+-------------+---------------------------------------------+---------------------------------------------+---------+------+---------+----------+-------------------------------------------------------------------------------------------+
| 1 | SIMPLE | article | NULL | index_merge | idx_article_processed,idx_article_taken | idx_article_processed,idx_article_taken | 2,2 | NULL | 6234059 | 100.00 | Using intersect(idx_article_processed,idx_article_taken); Using where; Using filesort |
+----+-------------+-----------+------------+-------------+---------------------------------------------+---------------------------------------------+---------+------+---------+----------+-------------------------------------------------------------------------------------------+
mysql> SELECT * FROM `judgement` where taken=0 and processed=0 ORDER BY updated_at asc limit 10;
+--------------------------------------+----------+-----------+---------------------------------------------------------------------------
| id | type | title | source| processed | released_at | created_at | updated_at | taken |
+--------------------------------------+----------+-----------+---------------------------------------------------------------------------
10 rows in set (9 min 15.97 sec)
taken, over both have indexes, why I put them together, the query get worse? shouldn't it be much faster due to more indexes?

Answer 1

I don't know an exact answer to the question "Why it become slow if the article table grows to 20 million rows".
Your query is doing two operations:
index_merge - Using intersect(idx_article_processed,idx_article_taken)
Using filesort
I only guess that up to 20 million rows in the table MySql can do both of these operations in the memory, but above this limit one of these operations (or maybe both) cannot fit in the memory buffer and MySql must use a file on the disk, which is much slower.
You can either increase memory buffers tweeking some MySql parameters or create indexes dedicated to your queries:
For this query:
SELECT * FROM `article` where taken=0 ORDER BY updated_at asc limit 10;
create this index:
CREATE INDEX my_new_index ON article( taken, updated_at )
For this query:
SELECT * FROM `article`
where taken=0 and over=0
ORDER BY updated_at asc limit 10;
create this index:
CREATE INDEX my_new_index1 ON article( taken, over, updated_at )
With help of these new indexes both filesort and megre operations will be eliminated.

Answer 2

The work involved in navigating an index gets worse than a table scan quite quickly. Yes/no indexing can be next to worthless if there's an even split.
If you've only got a few that match consider building another table for those rows concerned and join back, removing them when they are processed. In other dbs you'd build a conditional index.

Answer 3

It "became slow" because there are not that many rows with both taken=0 and over=0. And innodb_buffer_pool_size is too small. But, be careful, that setting should not be so large as to lead to swapping. How much RAM do you have available?