mariadb not using the correct index with subquery - mysql

Question

I'm facing a performance issue on a mariadb database. It seems to me that mariadb is not using the correct index when doing a request with a subquery, while injecting manually the result of the subquery in the request successfully uses the index:
Here is the request with bad behavior (note that the second subquery reads more rows than necessary):
ANALYZE SELECT `orders`.* FROM `orders`
WHERE `orders`.`account_id` IN (SELECT `accounts`.`id` FROM `accounts` WHERE `accounts`.`user_id` = 88144)
AND ( orders.type not in ("LimitOrder", "MarketOrder")
OR orders.type in ("LimitOrder", "MarketOrder") AND orders.state <> "canceled"
OR orders.type in ("LimitOrder", "MarketOrder") AND orders.state = "canceled" AND orders.traded_btc > 0 )
AND (NOT (orders.type = 'AdminOrder' AND orders.state = 'canceled')) ORDER BY `orders`.`id` DESC LIMIT 20 OFFSET 0 \G;
*************************** 1. row ***************************
id: 1
select_type: PRIMARY
table: accounts
type: ref
possible_keys: PRIMARY,index_accounts_on_user_id
key: index_accounts_on_user_id
key_len: 4
ref: const
rows: 7
r_rows: 7.00
filtered: 100.00
r_filtered: 100.00
Extra: Using index; Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: PRIMARY
table: orders
type: ref
possible_keys: index_orders_on_account_id_and_type,index_orders_on_type_and_state_and_buying,index_orders_on_account_id_and_type_and_state,index_orders_on_account_id_and_type_and_state_and_traded_btc
key: index_orders_on_account_id_and_type_and_state_and_traded_btc
key_len: 4
ref: bitcoin_central.accounts.id
rows: 60
r_rows: 393.86
filtered: 100.00
r_filtered: 100.00
Extra: Using index condition; Using where
When manually injecting the result of the subquery I have the correct behaviour (and expected performance):
ANALYZE SELECT `orders`.* FROM `orders`
WHERE `orders`.`account_id` IN (433212, 433213, 433214, 433215, 436058, 436874, 437950)
AND ( orders.type not in ("LimitOrder", "MarketOrder")
OR orders.type in ("LimitOrder", "MarketOrder") AND orders.state <> "canceled"
OR orders.type in ("LimitOrder", "MarketOrder") AND orders.state = "canceled" AND orders.traded_btc > 0 )
AND (NOT (orders.type = 'AdminOrder' AND orders.state = 'canceled'))
ORDER BY `orders`.`id` DESC LIMIT 20 OFFSET 0\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: orders
type: range
possible_keys: index_orders_on_account_id_and_type,index_orders_on_type_and_state_and_buying,index_orders_on_account_id_and_type_and_state,index_orders_on_account_id_and_type_and_state_and_traded_btc
key: index_orders_on_account_id_and_type_and_state_and_traded_btc
key_len: 933
ref: NULL
rows: 2809
r_rows: 20.00
filtered: 100.00
r_filtered: 100.00
Extra: Using index condition; Using where; Using filesort
1 row in set (0.37 sec)
Note that I have exactly the same issue when JOINing the two tables.
Here is an extract of the definitions of my orders table:
SHOW CREATE TABLE orders \G;
*************************** 1. row ***************************
Table: orders
Create Table: CREATE TABLE `orders` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`account_id` int(11) NOT NULL,
`traded_btc` decimal(16,8) DEFAULT '0.00000000',
`type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`state` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `index_orders_on_account_id_and_type_and_state_and_traded_btc` (`account_id`,`type`,`state`,`traded_btc`),
CONSTRAINT `orders_account_id_fk` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=8575594 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Does anyone know what's going on here?
Is there a way to force the database to use my index in my subrequest.

Answer 1

IN ( SELECT ... ) optimizes poorly. The usual solution is to turn into a JOIN:
FROM accounts AS a
JOIN orders AS o ON a.id = o.account_id
WHERE a.user_id = 88144
AND ... -- the rest of your WHERE
Or is that what you did with "Note that I have exactly the same issue when JOINing the two tables."? If so, let's see the query and it's EXPLAIN.
You refer to "expected performance"... Are you referring to the numbers in the EXPLAIN? Or do you have timings to back up the assertion?
I like to do this to get a finer grained look into how much "work" is going on:
FLUSH STATUS;
SELECT ...;
SHOW SESSION STATUS LIKE 'Handler%';
Those numbers usually make it clear whether a table scan was involved or whether the query stopped after OFFSET+LIMIT. The numbers are exact counts, unlike EXPLAIN, which is just estimates.
Presumably you usually look in orders via account_id? Here is a way to speed up such queries:
Replace the current two indexes
PRIMARY KEY (`id`),
KEY `account_id__type__state__traded_btc`
(`account_id`,`type`,`state`,`traded_btc`),
with these:
PRIMARY KEY (`account_id`, `type`, `id`),
KEY (id) -- to keep AUTO_INCREMENT happy.
This clusters all the rows for a given account, thereby making the queries run faster, especially if you are now I/O-bound. If some combination of columns makes a "natural" PK, then toss id completely.
(And notice how I shortened your key name without losing any info?)
Also, if you are I/O-bound, shrinking the table is quite possible by turning those lengthy VARCHARs (state & type) into ENUMs.
More
Given that the query involves
WHERE ... mess with INs and ORs ...
ORDER BY ...
LIMIT 20
and there are 2 million rows for that one user, there is no INDEX that can get past the WHERE to get into the ORDER BY so that it can consume the LIMIT. That is, it must perform this way:
filter through the 2M rows for that one user
sort (ORDER BY) some significant fraction of 2M rows
peel off 20 rows. (Yeah, 5.6 uses a "priority" queue, making the sort O(1) instead of O(log N), but this is not that much help.
I'm actually amazed that the IN( constants ) worked well.

Answer 2

I had the same problem. Please use inner join instead of in-subquery.