Optimising a MySQL Full-Text Query - mysql

Question

I have a table of medical diagnostic codes that users are able to perform a keyword search against. I have a column of descriptive text as well as a column of synonyms, both of which are considered. Results are presented in an auto-suggest format and the current implementation of the query is too slow for deployment:
SELECT
ID AS data, CONCAT('[', ICD10, '] ', description) AS value,
MAX(MATCH(description) AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) +
(MATCH(synonyms) AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) * 0.5)) AS relevance
FROM Code
WHERE
(MATCH(description) AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) OR
MATCH(synonyms) AGAINST ("fracture forearm current init oth" IN BOOLEAN MODE)) AND
isPCS = 0 AND
isEnabled = 1 AND
ICD10 IS NOT NULL AND
description IS NOT NULL
GROUP BY ID
ORDER BY relevance DESC
LIMIT 100
There are ~170K rows in the table, though the latter four static constraints reduce it to ~94K rows, of which ~16K have synonyms. A typical query takes 0.45 seconds on my desktop (i7-4770K) and about 0.75 seconds on our development server (a lower-end Xeon). Removing the ORDER BY keyword reduces it to 0.02 and 0.05 seconds, respectively.
I had expected that sorting the results would be trivial compared to the full-text search, but this doesn't appear to be the case. Am I missing a glaring inefficiency?
I'm also looking into eventually rebuilding this functionality on top of Lucene/Solr (opinions/suggestions welcomed), but I'd like to have a better understanding of this behaviour, and an optimised interim solution wouldn't hurt either.

Answer 1

If you order by relevance limit 100, it means MySQL has to find all rows that match your condition, evaluate your relevance formula, do a filesort, and take the first 100 of them.
If you don't order, it means MySQL has to find any 100 rows that fit the conditions, and can stop execution there.
So it is not the filesort after finding the result that makes it slow, it is that is has to find all results before doing the filesort (and there are probably a lot more than 100 rows that has at least some of the words you are looking for).
But there is actually an optimization you can use here: use a fulltext index on both of your columns together:
CREATE FULLTEXT INDEX idxft_Code_descr_syn ON Code (description, synonyms);
and then directly search in both columns together and order by the fulltext relevance directly without recalculating:
SELECT
ID AS data, CONCAT('[', ICD10, '] ', description) AS value,
MATCH(description, synonyms)
AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) AS relevance
FROM Code
WHERE
MATCH(description, synonyms)
AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) AND
isPCS = 0 AND
isEnabled = 1 AND
ICD10 IS NOT NULL AND
description IS NOT NULL
ORDER BY relevance
LIMIT 100
This will slightly change your relevance compared to your current order, because it will not weigh the synomym column differently than the description column, but since the result had been normalized for their own single column, your current weights may not have had the expected effect anyway.
The order by relevance will still require a full table search, but because of the way fulltext indexes work (they are supposed to order by the relevance), you will probably get a descent speedbump out of it (though any of your mentioned specialized search engines will be faster than a general purpose MySQL. If they are necessary for 170k rows is for you to test. More RAM might sometimes be worth a shot too. But that is an entirely different topic.)