I have a database of occupation titles I'm trying to run some queries on. I'm using Match() to try and find the best match occupational title for a user-entered string using this SQL:
SELECT *, MATCH (occupation_title) AGAINST ('EGG PROCESSOR')
AS score FROM occupational_titles WHERE MATCH (occupation_title)
AGAINST ('EGG PROCESSOR') ORDER BY score DESC;
When I run this query against my database, the first three results are "Processor", "Egg Processor", and "COPRA Processor". The first two have the exact same match score of 6.04861688613892. Why on earth would MySQL not rank an exact match hit as the number one result? Is there anything I can do to refine the search algorithm?
You probably want to use one of the modifier modes in your searches. Check the fulltext documentation.
In particular, by default it uses "natural language" searching, while you probably want to consider "boolean mode" and prefixing each keyword with a plus sign to make it mandatory in results, or using double quotes to search for the exact phrase. Check the boolean mode documentation for more information on the syntax.
You can also consider performing multiple searches using a variety of modes and doing your own weighting.
I guess you should change the collation of your Column to case insensitive ones.
eg. latin1 to latin1_bin
Case sensitive Match is being done in your case.
Have a look here:
http://dev.mysql.com/doc/refman/5.5/en/fulltext-natural-language.html
Related
I have strings like the following in my VARCHAR InnoDB table column:
"This is a {{aaaa->bbb->cccc}} and that is a {{dddd}}!"
Now, I'd like to search for e.g. {{xxx->yyy->zzz}}. Brackets are part of the string. Sometimes searched together with another colum, but which only contains an ordinary id and hence don't need to be considered (I guess).
I know I can use LIKE or REGEXP. But these (already tried) ways are too slow. Can I introduce a fulltext index? Or should I add another helping table? Should I replace the special characters {, }, -, > to get words for the fulltext search? Or what else could I do?
The search works with some ten-thousand rows and I assume that I often get about one hundred hits.
This link should give you all the info you need regarding FULLTEXT indexes in MySQL.
MySQL dev site
The section that you will want to pay particular attention to is:
"Full-text searching is performed using MATCH() ... AGAINST syntax. MATCH() takes a comma-separated list that names the columns to be searched. AGAINST takes a string to search for, and an optional modifier that indicates what type of search to perform. The search string must be a string value that is constant during query evaluation. This rules out, for example, a table column because that can differ for each row."
So in short, to answer your question you should see an improvement in query execution times by implementing a full text index on wide VARCHAR columns. Providing you are using a compatible storage engine ( InnoDB or MyISAM)
Also here is an example of how you can query the full text index and also an additional ID field as hinted in your question:
SELECT *
FROM table
WHERE MATCH (fieldlist) AGAINST ('search text here')
AND ( field2= '1234');
Let's say we have the following query:
SELECT *
FROM companies
WHERE name LIKE '%nited'
It returns
name
united
How do I write a query using MySQL's full-text search that will provide similar results?
Unfortunately you cannot do this using a MySQL full-text index. You cannot retrieve '*nited states' instantly from index because left characters are the most important part of the index. However, you can search 'United Sta*'.
// the only possible wildcard full-text search in MySQL
WHERE MATCH(column) AGAINST ('United Sta*' IN BOOLEAN MODE)
MySQL's full-text performs best when searching whole words in sentences - even that can suck at times. Otherwise, I'd suggest using an external full-text engine like Solr or Sphinx. I think Sphinx allows prefix and suffix wildcards, not sure about the others.
You could go back to MySQL's LIKE clause, but again, running queries like LIKE '%nited states' or LIKE '%nited Stat%', will also suffer on performance, as it can't use the index on the first few characters. 'United Sta%' and 'Unit%States' are okay as the index can be used against the first bunch of known characters.
Another quite major caveat using MySQL's full-text indexing is the stop-word list and minimum word length settings. For example, on a shared hosting environment, you will be limited to words greater than or equal to 4-characters. So searching 'Goo' to get 'Google' would fail. The stop-word list also disallows common words like 'and', 'maybe' and 'outside' - in-fact, there are 548 stop-words all together! Again, if not using shared hosting, these settings are relatively easily to modify, but if you are, then you will get annoyed with some of the default settings.
You can use MySQL's full-text index, but you must configure the parser to be the n-gram parser.
If your data is English (as opposed to Chinese or similar), you ought to also increase the ngram_token_size parameter to the minimum search term length you are willing to have. Otherwise, the search will be unacceptably slow.
You will also want to set innodb_ft_enable_stopword=0, otherwise an idiosyncrasy of how ngram stopword handling works will mean that many useful queries will return no results.
To explain why you must also increase ngram_token_size, you may think of this index as the following schema. MySQL then does a series of joins to find the results which match the search term:
CREATE TABLE fulltext_index
(
docid int(11) NOT NULL,
term char(2) NOT NULL,
PRIMARY KEY (docid, term),
INDEX term_idx (term)
);
The n-gram (2) parser breaks each word in your query into segments like se, eg, gm, me, en, nt, ts. For each of these n-grams, there are many results in English, so the index doesn't help much since it ends up iterating over everything anyway. Meanwhile, you can see how Chinese 随机的 would split into a much more useful 随机 and 机的. With n-gram size set to 4, the segments are segm, egme, gmen, ment, ents. These larger segments are much more likely to be unique, so each segment narrows down the search space significantly.
Disabling stopwords is also necessary because the ngram parser excludes all n-grams that contain any of the stopwords. For example, with an n-gram (4) parser, stopword will be parsed into stop, topw, opwr, pwor, and word:
stop will be excluded because it contains "to"
topw will be excluded because it contains "to"
opwr will be kept
pwor will be excluded because it contains "or"
word will be excluded because it contains "or"
Because these tokens are excluded from the index, a search for MATCH(name) AGAINST('stop' IN BOOLEAN MODE) would not return anything unless the stopwords mechanism is disabled before creating the index.
To answer your question,
set ngram_token_size to 3, 4, or whatever your minimum search term length is.
set innodb_ft_enable_stopword to 0 or OFF.
create the index with CREATE FULLTEXT INDEX companies_name_idx ON companies (name) WITH PARSER ngram;
SELECT * FROM companies WHERE MATCH(name) AGAINST('nited' IN BOOLEAN MODE);
This will also return results for nitedA, so you might want to further filter the results from there, if that's required for your application.
I'm trying to decide between two options to achieve a prefix match for names (50 Millions options) with MySQL. The usage is for an autocomplete for search results. The dilemma is between building:
An index on the VARCHAR and performing a LIKE 'word%' query
A FTS (full text search) index and performing a MATCH 'word*' query
Which is better for such a case? Should I consider additional FTS features for such an auto-suggest autocomplete of names?
FTS and prefix matching are two different things. So the answer depends on what your actual requirement is.
Do you need to return a list of all results that exactly match the condition column LIKE 'word%'? Specifically that the string must start with the word you're looking for.
Full text search does matching based on relevance. It's not always going to give you things that match specific strings. It does stemming, it has stopwords, it omits results if a word is too common.
I think in this case the best answer is "Full text search doesn't quite do what you think it does" So if you have precise requirements for matching, you should stick to the method that will work.
I'm recently doing some string searches from a table with about 50k strings in it, fairly large I'd say but not that big. I was doing some nested queries for a 'search within results' kinda thing. I was using LIKE statement to get a match of a searched keyword.
I came across MySQL's Full-Text search which I tried so I added a fulltext index to my str column. I'm aware that Full-text searches doesn't work on virtually created tables or even with Views so queries with sub-selects will not fit. I mentioned I was doing a nested queries, example is:
SELECT s2.id, s2.str
FROM
(
SELECT s1.id, s1.str
FROM
(
SELECT id, str
FROM strings
WHERE str LIKE '%term%'
) AS s1
WHERE s1.str LIKE '%another_term%'
) AS s2
WHERE s2.str LIKE '%a_much_deeper_term%';
This is actually not applied to any code yet, I was just doing some tests. Also, searching strings like this can be easily achieved by using Sphinx (performance wise) but let's consider Sphinx not being available and I want to know how this will work well in pure SQL query. Running this query on a table without Full-text added takes about 2.97 secs. (depends on the search term). However, running this query on a table with Full-text added to the str column finished in like 104ms which is fast (i think?).
My question is simple, is it valid to use LIKE or is it a good practice to use it at all in a table with Full-text added when normally we would use MATCH and AGAINST statements?
Thanks!
In this case you not neccessarily need subselects. You can siply use:
SELECT id, str
FROM item_strings
WHERE str LIKE '%term%'
AND str LIKE '%another_term%'
AND str LIKE '%a_much_deeper_term%'
... but also raises a good question: the order in which you are excluding the rows. I guess MySQL is smart enough to assume that the longest term will be the most restrictive, so starting with a_much_deeper_term it will eliminate most of the records then perform addtitional comparsion only on a few rows. - Contrary to this, if you start with term you will probably end up with many possible records then you have to compare them against the st of the terms.
The interesting part is that you can force the order in which the comparsion is made by using your original subselect example. This gives the opportunity to make a decision which term is the most restrictive based upon more han just the length, but for example:
the ratio of consonants a vowels
the longest chain of consonants of the word
the most used vowel in the word
...etc. You can also apply some heuristics based on the type of textual infomation you are handling.
Edit:
This is just a hunch but it could be possible to apply the LIKE to the words in the fulltext indexitself. Then match the rows against the index as if you have serched for full words.
I'm not sure if this is actually done, but it would be a smart thing to pull off by the MySQL people. Also note that this theory can only be used if all possible ocurrences arein fact in the fulltext search. For this you need that:
Your search pattern must be at least the size of the miimal word-length. (If you re searching for example %id% then it can be a part of a 3 letter word too, which is excluded by default form FULLTEXT index).
Your search pattern must not be a substring of any listed excluded word for example: and, of etc.
Your pattern must not contain any special characters.
I would like to use the position/index found by the Match...Against fulltext search in mysql to return some text before and after the match in the field. Is this possible? In all the examples I have seen, the Match...Against returns a score in the select instead of a location or position in the text field of which is being searched.
SELECT
random_field,
MATCH ($search_fields)
AGAINST ('".mysql_real_escape_string(trim($keywords))."' IN BOOLEAN MODE)
AS score
FROM indexed_sites
WHERE
MATCH ($search_fields)
AGAINST ('".mysql_real_escape_string($keywords)."' IN BOOLEAN MODE)
ORDER BY score DESC;
This will give me a field and a score...but I would like an index/position instead of (or along side) a score.
Fulltext searching is a scoring function. its not a search for occurrence function. In other words the highest scoring result may not have a starting position for the match. As it may be a combination of weighted results of different matches within the text. if you include query expansion the search for word/s may not even appear in the result!
http://dev.mysql.com/doc/refman/5.0/en/fulltext-query-expansion.html
I hope that makes some sense.
Anyway your best bet is to take the results and then use some text searching function to find the first occurrence of the first matching word. My guess is that would be best suited to a text processing language like perl or a more general language like php or what ever language you are using to run the query.
DC