Find matching questions from all records - mysql

Question

I've a library of questions and answers and building an API in NodeJS which allows to search for answers based on the question passed as input. Following is my goal:
Split the question by space
Tokenize it and remove stopwords
Query database for records where question contains one or more words from the tokenized array
Ideally sort in descending order total number of matches in the question. For eg: If the question A contains 'module' and 'solution' and question B contains only 'solution', then question A should be shown before question B
I've been able to achieve 1 to 3, using the below code:
let question = req.query.question;
let arrQuestions = question.split(" ");
let tokenizedQuestion = stopwords.removeStopwords(arrQuestions);
let whereClause = tokenizedQuestion.join("%' OR answer LIKE '%");
whereClause = " answer LIKE '%" + whereClause + "%' ";
let query = "SELECT * FROM tbl_libraries WHERE " + whereClause;
I'm not able to figure out how to achieve 4. Can somebody provide pointers?
Thanks!

Answer 1

Are you sure that you do not want to use MySQL fulltext search for this?
If the answer is some flavour of 'No', you can continue reading...
In one of my project I was implementing something like this.
Query wise it looks like this (simplified version):
SELECT
name
FROM
table
WHERE
name REGEXP 'term1|term2|term3' -- you can use your OR + LIKE way
ORDER BY
SP_TermsWeitght(name, 'term1 term2 term3') DESC
All the magic is in my SP_TermsWieght function that returns "weight" (number) and I'm supplying a list of terms (cleaned and normalized) to the function.
The function:
CREATE FUNCTION `SP_TermsWeight`(
`sValue` TEXT,
`sTerms` VARCHAR(127)
)
RETURNS INT
DETERMINISTIC
BEGIN
DECLARE i INT DEFAULT 1;
DECLARE p INT DEFAULT 1;
DECLARE w INT DEFAULT 0;
DECLARE l INT;
DECLARE c CHAR(1);
DECLARE s VARCHAR(63);
DECLARE delimiters VARCHAR(15) DEFAULT ' ,';
SET sTerms = TRIM(sTerms);
SET l = LENGTH(sTerms);
IF (l > 0) THEN
-- checking is value matched terms exactly
IF (sTerms = sValue) THEN
SET w = 50000;
ELSE
-- supposing that "the terms" is one single term so it it match in full, the weight will be high
IF (l <= 63) THEN
SET w = w + SP_TermWeight(sValue, sTerms, 5000, 1000, 100);
END IF;
-- not processing it term by term if it is already matched as full
IF (w = 0) THEN
-- processing term by term using space or comma as delimiter
WHILE i <= l DO
BEGIN
SET c = SUBSTRING(sTerms, i, 1);
IF (LOCATE(c, delimiters) > 0) THEN
SET s = SUBSTRING(sTerms, p, i - p);
SET w = w + SP_TermWeight(sValue, s, 50, 10, 0);
SET p = i + 1;
END IF;
SET i = i + 1;
END;
END WHILE;
IF (p > 1 AND p < i) THEN
SET s = SUBSTRING(sTerms, p, i - 1);
SET w = w + SP_TermWeight(sValue, s, 50, 10, 0);
END IF;
END IF;
END IF;
END IF;
RETURN w;
END
Technically speaking it is 'separating' terms using delimiters and checking if the value "contains" the term.
It's a bit hard to explain everything what it does (I've added a few comments in the code for you).
Feel free to ask questions if you do not understand some bits.
In your case it can be simplified dramatically as you do not need to differentiate begin/end/middle matches.
Another helper function that used internally:
CREATE FUNCTION `SP_TermWeight`(
`sValue` TEXT,
`sTerm` VARCHAR(63),
`iWeightBegin` INT,
`iWeightEnd` INT,
`iWeightMiddle` INT
)
RETURNS INT
DETERMINISTIC
BEGIN
DECLARE r INT DEFAULT 0;
SET sTerm = TRIM(sTerm);
IF (LENGTH(sTerm) > 1) THEN
IF (iWeightBegin != 0 AND sValue REGEXP CONCAT('[[:<:]]', sTerm)) THEN
SET r = r + iWeightBegin;
END IF;
IF (iWeightEnd != 0 AND sValue REGEXP CONCAT(sTerm, '[[:>:]]')) THEN
SET r = r + iWeightEnd;
END IF;
IF (r = 0 AND iWeightMiddle != 0 AND sValue REGEXP sTerm) THEN
SET r = r + iWeightMiddle;
END IF;
END IF;
RETURN r;
END
This function used for assigning different weights if the term matched to value from the beginning of the string, at the end of the string or in the middle. It is important in my case. In your case it might be simple LIKE.

Answer 2

I ended up using Full Text Search. Following is the stored procedure I created to enable searching:
DROP PROCEDURE IF EXISTS SP_Search $$
CREATE PROCEDURE `SP_Search`(IN QuestionToSearch TEXT, IN TagsToSearch TEXT, IN CollectionsToSearch TEXT, IN ReturnRecordsFromIndex INT, IN TotalRecordsToReturn INT)
BEGIN
SET #MainQuery = CONCAT("SELECT *, MATCH(question, answer_content) AGAINST (", CONCAT("'", QuestionToSearch, "'"), " IN NATURAL LANGUAGE MODE) AS score ");
SET #MainQuery = CONCAT(#MainQuery, " FROM tbl_libraries ");
SET #MainQuery = CONCAT(#MainQuery, " WHERE MATCH(question, answer_content) AGAINST (", CONCAT("'", QuestionToSearch, "'"), " IN NATURAL LANGUAGE MODE) ");
IF F_IsNullOrEmpty(TagsToSearch) AND NOT F_IsNullOrEmpty(CollectionsToSearch) THEN
SET #MainQuery = CONCAT(#MainQuery, " AND collections LIKE '%", CollectionsToSearch, "%' ");
ELSEIF F_IsNullOrEmpty(CollectionsToSearch) AND NOT F_IsNullOrEmpty(TagsToSearch) THEN
SET #MainQuery = CONCAT(#MainQuery, " AND tags LIKE '", TagsToSearch, "' ");
ELSEIF NOT F_IsNullOrEmpty(TagsToSearch) AND NOT F_IsNullOrEmpty(CollectionsToSearch) THEN
SET #MainQuery = CONCAT(#MainQuery, " AND tags LIKE '", TagsToSearch, "' AND collections LIKE '", CollectionsToSearch, "' ");
END IF;
SET #MainQuery = CONCAT(#MainQuery, " ORDER BY score DESC ");
SET #MainQuery = CONCAT(#MainQuery, " LIMIT ", ReturnRecordsFromIndex, ", ", TotalRecordsToReturn);
PREPARE SqlQuery FROM #MainQuery;
EXECUTE SqlQuery;
END $$
DELIMITER ;
This uses a custom function I created F_IsNullOrEmpty, which is as shown below for completion:
CREATE FUNCTION F_IsNullOrEmpty(ValueToCheck VARCHAR(256)) RETURNS BOOL
DETERMINISTIC
BEGIN
IF((ValueToCheck IS NULL) OR (LENGTH(ValueToCheck) = 0) OR (ValueToCheck = 'null')) THEN
Return True;
ELSE
Return False;
END IF;
END;