mysql ngrams indexing example - mysql

Question

As I have read in many parts, ngram indexing can improve word searches.
In this old post it says that it could be adapted for mysql but it does not say how: levenshtein alternative
Can anyone put some example as you can use this technique in mysql?
Can use this technique to improve the performance of the levenshtein function for mysql?.
My need is to find approximate text (like levenshtein)
In mi tests I used levenshtein() and levenshtein_ratio() functions from:
http://www.artfulsoftware.com/infotree/qrytip.php?id=552
SELECT *, levenshtein_ratio('stacoverflou',words_column) AS ratio
FROM my_table
ORDER BY ratio DESC
That improve performance (Supposing not misspelled first letter)
SELECT *, levenshtein_ratio('stacoverflou',words_column) AS ratio
FROM my_table
WHERE words_column LIKE 's%'
ORDER BY ratio DESC
Also I found this php library for building ngrams:
https://gist.github.com/Xeoncross/5366393
But I have no idea how to use these ngrams in mysql

Answer 1

I have finally made an algorithm myself:
Generate ngrams algorithm:
I build a words table with 3 columns: ngrams (fulltext), word (UNIQUE), lang
I used Bigrams() function to make ngrams for each word
I added a char padding to each ngram to skip the full text index minimum word length ex: 'abcd' should be 'ab bc cd' but with padding it look like 'abxx bcxx cdxx'
Search algorithm:
I take the client written words as correct and use them to search in the real table
with mysql fulltext search query using AGAINST('+word_1 +word_2 +word_n' IN BOOLEAN MODE)
if the score (ranking) > 0 Mission accomplished and ngrams not used (show result to client)
If the score (ranking) = 0 (maybe misspelled words) then use ngrams words table to retrieve correct words
Retrieve correct words from ngrams algorithm:
For each word generate ngrams and preform a fulltext search query using AGAINST('abxx bcxx cdxx' IN BOOLEAN MODE) FROM words table (where we have the ngrams column) and retrieve the correct word. Note that here no (+) prefix to ngrams
Rebuild the search like as step 4
If score > 0 Mission accomplished -> show results -> END
If score still 0 then make another query but this time without (+) +word prefix and IN NATURAL LANGUAGE MODE -> show results -> END
Step 2 code:
// original from : https://gist.github.com/Xeoncross/5366393
// modified for working also with unicode characters
function Bigrams($word){
$ngrams = array();
$len = mb_strlen($word);
for($i=0;$i+1<$len;$i++){
$ngram = mb_substr($word, $i, 2);
while(mb_strlen($ngram) < 4){
$ngram .= "x";
}
$ngrams[$i]=$ngram;
}
return implode(" ",$ngrams);
}
Step 4 code:
SELECT my_column,
( MATCH(my_full_text_column )
AGAINST('+word_1 +word_2 +word_n' IN BOOLEAN MODE)
) AS score LIMIT 10
Step 7 code:
$word = "stacoverflou"; // Intentionally misspelled
$actual_word_ngrams = Bigrams();
//that return
//stxx taxx acxx coxx ovxx vexx erxx rfxx flxx loxx ouxx
SELECT word,
( MATCH( ngrams )
AGAINST('$actual_word_ngrams' IN BOOLEAN MODE)
) AS score LIMIT 1
That return stackoverflow an can be used with ohers words (if many) to much more accurate search like as step 4
END