Getting a user's position in a queue table is too slow - mysql

Question

I have a MariaDB table that looks like this:
+--------+--------+--------+---------------------+
| realm | key2 | userId | date |
+--------+--------+--------+---------------------+
| AB3 | 123 | 1 | 2017-08-04 17:30:00 |
| AB3 | 124 | 1 | 2017-08-04 17:30:00 |
| AB3 | 125 | 1 | 2017-08-04 17:30:00 |
| XY7 | 97 | 2 | 2017-08-04 17:35:00 |
| XY7 | 98 | 2 | 2017-08-04 17:35:00 |
| XY7 | 99 | 2 | 2017-08-04 17:35:00 |
| AB3 | 110 | 3 | 2017-08-04 17:40:00 |
| AB3 | 111 | 3 | 2017-08-04 17:40:00 |
+--------+--------+--------+---------------------+
PRIMARY_KEY (realm, key2)
INDEX (realm, userId)
INDEX (date)
This table operates as some sort of queue for processing user actions. Basically a server always takes the oldest data from this table, processes it and deletes it from this table. Each realm has its own server processing this queue.
Now I want to find out a user's position in queue for that realm. So, using the example above, when I request the position for userId 3 in realm 'AB3', I want to get the result 2 because only one other user (userId 1) is to be processed earlier for realm AB3.
(The row key2 might be irrelevant in this example. I only included it because it is part of the primary key which may make it relevant for finding a good solution)
Here is the SQL schema:
CREATE TABLE `queue` (
`realm` varchar(5) NOT NULL,
`key2` int(10) UNSIGNED NOT NULL,
`userId` int(10) UNSIGNED NOT NULL,
`date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
INSERT INTO `queue` (`realm`, `key2`, `userId`, `date`) VALUES
('AB3', 110, 3, '2017-08-04 17:40:00'),
('AB3', 111, 3, '2017-08-04 17:40:00'),
('AB3', 123, 1, '2017-08-04 17:30:00'),
('AB3', 124, 1, '2017-08-04 17:30:00'),
('AB3', 125, 1, '2017-08-04 17:30:00'),
('XY7', 97, 2, '2017-08-04 17:35:00'),
('XY7', 98, 2, '2017-08-04 17:35:00'),
('XY7', 99, 2, '2017-08-04 17:35:00');
ALTER TABLE `queue`
ADD PRIMARY KEY (`realm`,`key2`),
ADD KEY `ru` (`realm`,`userId`) USING BTREE,
ADD KEY `date` (`date`);
I came up with this query that seems to work but is pretty slow (~3 seconds) on a table with 10,000,000 entries:
SELECT (COUNT(DISTINCT `realm`, `userId`)+1) `position`
FROM `queue`
WHERE `realm` = 'AB3'
AND `date` < (
SELECT `date`
FROM `queue`
WHERE `realm` = 'AB3' AND `userId` = 3
GROUP BY `realm`, `userId`
)
SQL Fiddle: http://sqlfiddle.com/#!9/fb04fd/9/0
EXPLAIN EXTENDED of this query:
+----+-------------+-------+-------------+-----------------+------------+---------+-------+---------+----------+------------------------------------------+--+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | |
+----+-------------+-------+-------------+-----------------+------------+---------+-------+---------+----------+------------------------------------------+--+
| 1 | PRIMARY | queue | ref | PRIMARY,ru,date | PRIMARY | 767 | const | 5266123 | 100.00 | Using where | |
| 2 | SUBQUERY | queue | index_merge | PRIMARY,ru | ru,PRIMARY | 771,767 | | 496 | 75.00 | Using intersect(ru,PRIMARY); Using where | |
+----+-------------+-------+-------------+-----------------+------------+---------+-------+---------+----------+------------------------------------------+--+
Do you have any ideas how I can optimize this query to run faster on a table with like 10,000,000 entries?
Other queries that are run on this table:
SELECT `m`.*
FROM `queue` `m`
JOIN (
SELECT `m`.*
FROM `queue` `m`
WHERE `m`.`realm` = ?
ORDER BY `date` ASC
LIMIT 1
) `mm` ON `m`.`realm` = `mm`.`realm` AND `m`.`userId` = `mm`.`userId`;
and
DELETE FROM `queue` WHERE `realm` = ? AND `userId` = ?;
How could I optimize my indexes?

Answer 1

I feel like something wrong with the table DDL. Anyway, i would have rewriten your query like :
SELECT (COUNT(DISTINCT `userId`)+1) `position`
FROM `queue`
WHERE `realm` = 'AB3'
AND `date` < (
SELECT min(`date`)
FROM `queue`
WHERE `realm` = 'AB3' AND `userId` = 3
)
and perhaps have a really specific index for this query like :
index (realm, date)
You can try the sheety index
index (realm, date, userId)
but not even sure it will be faster that the previous one.