Indexing for keyset pagination in mysql - mysql

Question

I am trying to build an index in mysql to support a keyset pagination query. My query looks like this:
SELECT * FROM invoice
WHERE company_id = 'someguid'
AND id > 'lastguidfromlastpage'
ORDER BY id
LIMIT 10
Common knowledge on this says an index on company_id would contain the PRIMARY KEY of the table (id). Because of this I would expect to be able to use rows directly from the index without any need for the query to sort results first however my explain plan shows a filesort and an index merge:
mysql> explain SELECT *
-> FROM invoice
-> WHERE company_id = '37687714-2e9d-4daa-aee6-f7d56962f903'
-> AND id > '525ae038-0cc3-4f9a-85e6-6f36d43fae40'
-> ORDER BY id
-> LIMIT 10;
+----+-------------+---------+------------+-------------+-----------------------------+-----------------------------+---------+------+------+----------+---------------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+-------------+-----------------------------+-----------------------------+---------+------+------+----------+---------------------------------------------------------------------------+
| 1 | SIMPLE | invoice | NULL | index_merge | PRIMARY,invoice__company_id | invoice__company_id,PRIMARY | 76,38 | NULL | 48 | 100.00 | Using intersect(invoice__company_id,PRIMARY); Using where; Using filesort |
+----+-------------+---------+------------+-------------+-----------------------------+-----------------------------+---------+------+------+----------+---------------------------------------------------------------------------+
1 row in set, 1 warning (0.00 sec)
If I explicitly add the id to the index then I get the explain plan I would expect:
mysql> explain SELECT *
-> FROM invoice
-> WHERE company_id = '37687714-2e9d-4daa-aee6-f7d56962f903'
-> AND id > '525ae038-0cc3-4f9a-85e6-6f36d43fae40'
-> ORDER BY id
-> LIMIT 10;
+----+-------------+---------+------------+-------+--------------------------------+--------------------------------+---------+------+------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+-------+--------------------------------+--------------------------------+---------+------+------+----------+-----------------------+
| 1 | SIMPLE | invoice | NULL | range | PRIMARY,invoice__company_id_id | invoice__company_id_id,PRIMARY | 76 | NULL | 98 | 100.00 | Using index condition |
+----+-------------+---------+------------+-------+--------------------------------+--------------------------------+---------+------+------+----------+-----------------------+
1 row in set, 1 warning (0.00 sec)
SHOW CREATE TABLE:
CREATE TABLE `invoice` (
`id` varchar(36) NOT NULL,
`company_id` varchar(36) NOT NULL DEFAULT '0',
`invoice_number` varchar(36) NOT NULL DEFAULT '0',
`identifier` varchar(255) NOT NULL,
`created_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`created_by` varchar(36) DEFAULT NULL,
`data_source` varchar(36) NOT NULL,
`type` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `invoice__company_id_id` (`company_id`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
select ##optimizer_switch;
use_index_extensions=on
MySQL version:
version: 5.7.26-29-57-log
innodb_version: 5.7.26-29
version_comment: Percona XtraDB Cluster (GPL), Release rel29, Revision 03540a3, WSREP version 31.37, wsrep_31.37
SHOW VARIABLES LIKE 'char%';
character_set_client utf8
character_set_connection utf8
character_set_database latin1
character_set_filesystem binary
character_set_results utf8
character_set_server latin1
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
There are a few sources explaining that the company_id index on it's own should be sufficient for this:
https://stackoverflow.com/a/30152513/64023
https://dba.stackexchange.com/a/136029/166838
I've been unable to find official documentation about exactly what to expect. Is this related to the datatypes for the id? Is the common knowledge about mysql+innodb behavior incorrect?

Answer 1

I have encountered this problem before. Here is my analysis of it.
It occurs in MySQL 5.7 and 8.0, but apparently not in older versions and not in MariaDB.
The "solution" I prefer is to change the indexes thus:
INDEX(company_id) -- DROP this
INDEX(company_id, id) -- ADD this
Although the 2-column index is theoretically identical to the one-column index for InnoDB (assuming id is the PK`), the Optimizer seems to ignore this fact in some situations.
Also, I like to explicitly add the PK when I see a need. This signals future readers of the schema (including myself) that some query benefits from the PK being appended.
I have yet to find a case where "index merge intersect" is faster than an equivalent composite index.
I dislike ever using index "hints" for fear that the data distribution will change in the future and my "hint" will make things worse.

Answer 2

This won't work.
For keyset pagination to take effect, you need to have autoincrement integer as your primary id/key. Right now you are using VARCHAR and store UIDs.
Your query won't select "next" UID "larger than" (... AND id > '525ae038-0cc3-4f9a-85e6-6f36d43fae40' ... ).
When you change primary ID to be number, then, this will work.
If you still have issues with indexes, you can try forcing mysql to use your index:
SELECT * FROM invoice USE INDEX (invoice__company_id_id)
WHERE company_id = 'someguid'
AND id > 12345
ORDER BY id
LIMIT 10