I've been looking for resources to explain how this query exactly sorts retrieved items by relevance, and haven't been able to find any.
Hopefully one of you can explain the logistics of it to me?
SELECT *, MATCH(body, subject) AGAINST ('words' IN BOOLEAN MODE) AS relevance
FROM `messages`
WHERE MATCH(body, subject) AGAINST ('words' IN BOOLEAN MODE)
ORDER BY relevance DESC
In this case, I know that first half of this query searches through the messages.body and messages.subject columns for the search terms "words". It then returns those results, (regardless of the Boolean Operators) in what is essential a "random order" (ordered by what is found first, then found 2nd, and so on).
What I don't understand, however, is how MySQL interprets the WHERE clause and the rest of the query. How does repeating the first half of code reorder the results by relevance?
For example, an ORDER BY clause that sorts a users.user_id column by desc. numerical order MAKES SENSE to me because each row/cell has a clear order (e.g. - 3 , 2 , 1, and so on)
But how does (going back to the original query) MySQL interpret these "word" results (words, obviously not having any values/numbers/clear-order) and sort them according to relevance?
Is it because the Boolean Full-text Search gives hidden numerical values to these search terms? Like if the AGAINST clause read:
AGAINST ('+apple -macintosh ~microsoft >windows' IN BOOLEAN MODE)
Like "apple" gets a value of 100, "macintosh" a value of -100, "microsoft" a value of 20, and "windows" a value of 40 (to reflect the Operator Effects)?
I know that this is oversimplifying the process (especially when considering if a column contains more than one of these search terms), but that is the best I got.
What I basically need, is a layman-terms explanation of the WHERE clause's (the 2nd half of query code's) effect on the query results as a whole.
Related
Im currently using a query for an autocomplete box with like. However I want to use the match, against which should be faster but I'm running against some issues with the sorting.
I want to rank a query like this:
[query] %
[query]%
% [query]%
%[query]%
For now I use
SELECT * FROM table
WHERE name LIKE '%query%'
ORDER BY (case
WHEN name LIKE 'query %' THEN 1
WHEN name LIKE 'query%' THEN 2
WHEN name LIKE '% query%' THEN 3
ELSE 4 END) ASC
When I use...
SELECT * FROM table
WHERE MATCH(name) AGAINST('query*' IN BOOLEAN MODE)
...all results get the same 'ranking score'.
For example searching for Natio
returns Pilanesberg National Park and National Park Kruger with the same score while I want the second result as first becouse it starts with the query.
How can I achieve this?
I had your same problem and I had to approach it in a different way.
The documentation of MySQL says:
The term frequency (TF) value is the number of times that a word appears in a document. The inverse document frequency (IDF) value of a word is calculated using the following formula, where total_records is the number of records in the collection, and matching_records is the number of records that the search term appears in.
${IDF} = log10( ${total_records} / ${matching_records} )
When a document contains a word multiple times, the IDF value is multiplied by the TF value:
${TF} * ${IDF}
Using the TF and IDF values, the relevancy ranking for a document is calculated using this formula:
${rank} = ${TF} * ${IDF} * ${IDF}
And this is followed by an example where it explains the above declaration: it search for the word 'database' in different fields and returns a rank based upon the results.
In your example the words "Pilanesberg National Park", "National Park Kruger" will return the same rank against ('Natio' IN BOOLEAN MODE)* because the rank is based not on the common sense similarity of the word (or in this case you'd expected to tell the database what's meaning -for you- "similar to"), but is based on the above formula, related to the frequency.
And note also that the value of the freqency is affected by the type of index (InnoDB or MyISAM) and by the version of MySQL (in older version you cannot use Full-text indexes with InnoDB tables).
Regarding your problem, you can use MySQL user defined variables or functions or procedures in order to evaluate the rank basing upon your idea of rank. Examples here, here or here. And also here.
See also:
MySQL match() against() - order by relevance and column?
MYsql FULLTEXT query yields unexpected ranking; why?
I have a table of medical diagnostic codes that users are able to perform a keyword search against. I have a column of descriptive text as well as a column of synonyms, both of which are considered. Results are presented in an auto-suggest format and the current implementation of the query is too slow for deployment:
SELECT
ID AS data, CONCAT('[', ICD10, '] ', description) AS value,
MAX(MATCH(description) AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) +
(MATCH(synonyms) AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) * 0.5)) AS relevance
FROM Code
WHERE
(MATCH(description) AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) OR
MATCH(synonyms) AGAINST ("fracture forearm current init oth" IN BOOLEAN MODE)) AND
isPCS = 0 AND
isEnabled = 1 AND
ICD10 IS NOT NULL AND
description IS NOT NULL
GROUP BY ID
ORDER BY relevance DESC
LIMIT 100
There are ~170K rows in the table, though the latter four static constraints reduce it to ~94K rows, of which ~16K have synonyms. A typical query takes 0.45 seconds on my desktop (i7-4770K) and about 0.75 seconds on our development server (a lower-end Xeon). Removing the ORDER BY keyword reduces it to 0.02 and 0.05 seconds, respectively.
I had expected that sorting the results would be trivial compared to the full-text search, but this doesn't appear to be the case. Am I missing a glaring inefficiency?
I'm also looking into eventually rebuilding this functionality on top of Lucene/Solr (opinions/suggestions welcomed), but I'd like to have a better understanding of this behaviour, and an optimised interim solution wouldn't hurt either.
If you order by relevance limit 100, it means MySQL has to find all rows that match your condition, evaluate your relevance formula, do a filesort, and take the first 100 of them.
If you don't order, it means MySQL has to find any 100 rows that fit the conditions, and can stop execution there.
So it is not the filesort after finding the result that makes it slow, it is that is has to find all results before doing the filesort (and there are probably a lot more than 100 rows that has at least some of the words you are looking for).
But there is actually an optimization you can use here: use a fulltext index on both of your columns together:
CREATE FULLTEXT INDEX idxft_Code_descr_syn ON Code (description, synonyms);
and then directly search in both columns together and order by the fulltext relevance directly without recalculating:
SELECT
ID AS data, CONCAT('[', ICD10, '] ', description) AS value,
MATCH(description, synonyms)
AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) AS relevance
FROM Code
WHERE
MATCH(description, synonyms)
AGAINST("fracture forearm current init oth" IN BOOLEAN MODE) AND
isPCS = 0 AND
isEnabled = 1 AND
ICD10 IS NOT NULL AND
description IS NOT NULL
ORDER BY relevance
LIMIT 100
This will slightly change your relevance compared to your current order, because it will not weigh the synomym column differently than the description column, but since the result had been normalized for their own single column, your current weights may not have had the expected effect anyway.
The order by relevance will still require a full table search, but because of the way fulltext indexes work (they are supposed to order by the relevance), you will probably get a descent speedbump out of it (though any of your mentioned specialized search engines will be faster than a general purpose MySQL. If they are necessary for 170k rows is for you to test. More RAM might sometimes be worth a shot too. But that is an entirely different topic.)
How can I do a MySQL search which will match partial words but also provide accurate relevancy sorting?
SELECT name, MATCH(name) AGAINST ('math*' IN BOOLEAN MODE) AS relevance
FROM subjects
WHERE MATCH(name) AGAINST ('math*' IN BOOLEAN MODE)
The problem with boolean mode is the relevancy always returns 1, so the sorting of results isn't very good. For example, if I put a limit of 5 on the search results the ones returned don't seem to be the most relevant sometimes.
If I search in natural language mode, my understanding is that the relevancy score is useful but I can't match partial words.
Is there a way to perform a query which fulfils all of these criteria:
Can match partial words
Results are returned with accurate relevancy
Is efficient
The best I've got so far is:
SELECT name
FROM subjects
WHERE name LIKE 'mat%'
UNION ALL
SELECT name
FROM subjects
WHERE name LIKE '%mat%' AND name NOT LIKE 'mat%'
But I would prefer not to be using LIKE.
The new InnoDB full-text search feature in MySQL 5.6 helps in this case.
I use the following query:
SELECT MATCH(column) AGAINST('(word1* word2*) ("word1 word1")' IN BOOLEAN MODE) score, id, column
FROM table
having score>0
ORDER BY score
DESC limit 10;
where ( ) groups words into a subexpression. The first group has like word% meaning; the second looks for exact phrase. The score is returned as float.
I obtained a good solution in this (somewhat) duplicate question a year later:
MySQL - How to get search results with accurate relevance
Well i'm running 2 queries that should show me the same result,
First query:
SELECT count( id ) AS cv FROM table_name WHERE field_name LIKE '%êêê01, word02, word03%'
Second query:
SELECT count( id ) AS cv FROM table_name WHERE match(field_name) against('êêê01, word02, word03')
but the first show more rows than the second, someone could tell me why?
I'm using fulltext index on this field,
Thanks.
I did a quick research and the following quote should answer your question:
One problem with MATCH on MySQL is that it seems to only match against whole words so a search for 'bla' won't match a column with a value of 'blah'.
It's also described in the documentation for match
By default, the MATCH() function performs a natural language search for a string against a text collection. A collection is a set of one or more columns included in a FULLTEXT index. The search string is given as the argument to AGAINST(). For each row in the table, MATCH() returns a relevance value; that is, a similarity measure between the search string and the text in that row in the columns named in the MATCH() list.
Meanwhile like is more "powerful" as it can look upon individuals characters:
Per the SQL standard, LIKE performs matching on a per-character basis, thus it can produce results different from the = comparison operator:
Which explains why like returns more results than match.
I have a table with names of movies, and I want to be able to search for a movie in that table. But I want to be able to search for part of the title, and still return a result. For example, if there is a record with the name "The quantum of solace", then I want to be able to do a search for "quantum solace", or even "007: quantum solace" and I want to find that record. Is there a way to do this?
EDIT
And how do I sort according to the matches? That is, the row that matches the most, should be returned first.
Use a MySQL Full Text Search in boolean mode.
If you do this when you search for '007: quantum solace' as it contains at least one matching result in the column it will be displayed, you can then order by relevancy.
SELECT *, MATCH(title) AGAINST ('quantum solace' IN BOOLEAN MODE) AS rank
FROM films
WHERE MATCH(title) AGAINST ('quantum solace' IN BOOLEAN MODE) ORDER BY rank DESC
Have a look at the full text search capabilities of MySQL. Once you set up a full text index, you can do queries like this one:
SELECT *
FROM movies
WHERE MATCH(title) AGAINST ('quantum solace' IN BOOLEAN MODE)