Our DB contains a lot of entries with comma in their titles (as in Hungarian, it's the decimal separator instead of period) and would like to match those with the right relevance. The search SQL looks like this currently in case of the user inputted terms are 7,5x20 otherTerm
SELECT (MATCH(title) AGAINST('(+7,5x20* +otherTerm* ) (7,5x20* otherTerm* ) (+7,5x20 +otherTerm )' IN BOOLEAN MODE)) AS Relevance,
id, title, product_id FROM versions
WHERE (MATCH(title) AGAINST('(+7,5x20* +otherTerm* ) (7,5x20* otherTerm* ) (+7,5x20 +otherTerm )' IN BOOLEAN MODE))
ORDER BY Relevance DESC LIMIT 50
Now the result order gives a higher relevance for eg. 5x20 than 7,5x20 so there has to be some kind of character escaping has to be done on the comma, preventing MySQL to handle them as separated strings. I didn't find the right one.
Thanks for any help in advance.
Edit: disassembling title into more digestible data is currently not an option. Really looking for solution escaping the comma or replacing it with 'match any single character' operator like dot in regex.
FULLTEXT indexing is not designed to handle numbers, regardless of the Locale for the numbers.
One approach is to alter the incoming text to replace punctuation that you want to treat as "letters" into, say, _. (And build a separate column for storing this altered text into. Then add the FULLTEXT index to it instead of the "real" text.)
Please note that +x will fail in a bad way -- one character strings are not indexed, so they cannot be found. So including strings that are two short leads to returning zero matches.
Alterations to the saved text (eg, 7_5x20) needs to be applied to the search, too.
50K rows? Write a special, one-time, script to perform the above transformation to the existing 50K rows. Then incorporate the transformations into both the INSERTs and the SELECTs.
Related
I had done several different searches on SO looking for a simple solution to sorting mySQL results alphanumerically where some fields may have special characters present. The solution:
"SELECT *, REPLACE(title '\"', '') AS indexTitle ORDER BY indexTitle ASC";
In this case I'm searching for strings that begin with a double quote, escaped.
This probably wouldn't be a great solution where the types of special characters are not known, but for a simple sort it works nicely.
Hopefully this helps someone.
One way to do this would be to write your own function to strip non-alphanumeric characters from a String. Google found me this example (I've not checked it!). Then you could write something like:
SELECT *, remove_non_alphanum_char_f(title) AS indexTitle ORDER BY indexTitle ASC;
Though of course as #arkascha has pointed out in the comments above this is slow and not scalable. A better solution is to go back a step and, if possible, ensure the data in your table is in the correct format to begin with. If you really need the special characters, it may be less of an overhead to add an extra column to your table which is the title column with the special characters stripped - then you could just order by that column. You could perform the stripping at the point when you insert into the table.
SELECT user_id, user_name.fullname, live, likes,
MATCH (fullname, email, live) AGAINST (:search_I IN BOOLEAN MODE) AS relevance
FROM profile LEFT JOIN user_name ON user_id=user_id
WHERE MATCH (fullname, email, live) AGAINST (:search_II IN BOOLEAN MODE)
ORDER BY relevance DESC
bindValue(':search_I', $search...);
bindValue(':search_II', $search...);//PDO can't use same one twice
I have a query use FULL TEXT search, I need to order by the closest match on top.
However this query is not working, It didn't order anything.
I did a testing, search 123#hotmail.com
2 rows in my db, abc#hotmail.com & 123#hotmail.com
It return 2 rows but didn't put the closest match on top(123#hotmail.com)
anyone know where is the problems?
By default MySQL full text search has a minimum word length of 3 (see here).
So, your example of '123#hotmail.com' is only matching on 'hotmail' and the two are equivalent.
You can change the default (and rebuild the index). But, I'd suggest that you do testing with 'abcd#hotmail.com' instead.
EDIT:
The definition of a word is buried a bit in the documentation:
The MySQL FULLTEXT implementation regards any sequence of true word
characters (letters, digits, and underscores) as a word. That sequence
may also contain apostrophes (“'”), but not more than one in a row.
This means that aaa'bbb is regarded as one word, but aaa''bbb is
regarded as two words. Apostrophes at the beginning or the end of a
word are stripped by the FULLTEXT parser; 'aaa'bbb' would be parsed as
aaa'bbb.
Because of the where clause, you can see that there is a match to both email addresses. That match would have to be on 'hotmail'. The 'com' and email name get chopped off because of the default minimum word length.
I'm pretty new to MySQL full-text searches and I ran into this problem today:
My company table has a record with "e-magazine AG" in the name column. I have a full-text index on the name column.
When I execute this query the record is not found:
SELECT id, name FROM company WHERE MATCH(name) AGAINST('+"e-magazi"*' IN BOOLEAN MODE);
I need to work with quotes because of the dash and to use the wildcard because I implement a "search as you type" functionality.
When I search for the whole term "e-magazine AG", the record is found.
Any ideas what I'm doing wrong here? I read about adding the dash to the list of word characters (config update needed) but I'm searching for a way to do this programmatically.
This clause
MATCH(name) AGAINST('+"e-magazi"*' IN BOOLEAN MODE);
Will search for a AND "e" AND NOT "magazi"; i.e. the - inside "e-magazi" will be interpreted as a not even though it is inside quotation marks.
For this reason it will not work as expected.
A solution is to apply an extra having clause with a LIKE.
I know this having is slow, but it will only be applied to the results of the match, so not too many rows should be involved.
I suggest something like:
SELECT id, name
FROM company
WHERE MATCH(name) AGAINST('magazine' IN BOOLEAN MODE)
HAVING name LIKE '%e-magazi%';
MySQL fulltext treats the word e-magazine in a text as a phrase and not as a word. Because of that it results the two words e and magazine. And while it builds the search index it does not add the e to the index because of the ft_min_word_len (default is 4 chars).
The same length limitation is used for the search query. That is the reason why a search for e-magazine returns exactly the same results as a-magazine because a and - is fully ignored.
But now you want to find the exact phrase e-magazine. By that you use the quotes and that is the complete correct way to find phrases, but MySQL does not support operators for phrases, only for words:
https://dev.mysql.com/doc/refman/5.7/en/fulltext-boolean.html
With this modifier, certain characters have special meaning at the beginning or end of words in the search string
Some people would suggest to use the following query:
SELECT id, name
FROM company
WHERE MATCH(name) AGAINST('e-magazi*' IN BOOLEAN MODE)
HAVING name LIKE 'e-magazi%';
As I said MySQL ignores the e- and searches for the wildcard word magazi*. After those results are optained it uses HAVING to aditionally filter the results for e-magazi* including the e-. By that you will find the phrase e-magazine AG. Of course HAVING is only needed if the search phrase contains the wildcard operator and you should never use quotes. This operator is used by your user and not you!
Note: As long you do not surround the search phrase with % it will find only fields that start with that word. And you do not want to surround it, because it would find bee-magazine as well. So maybe you need an additional OR HAVING name LIKE ' %e-magazi%' OR HAVING NAME LIKE '\\n%e-magazi%' to make it usable inside of texts.
Trick
But finally I prefer a trick so HAVING isn't needed at all:
If you add texts to your database table, add them additionally to a separate fulltext indexed column and replace words like up-to-date with up-to-date uptodate.
If a user searches for up-to-date replace it in the query with uptodate.
By that you can still find specific in user-specific but up-to-date as well (and not only date).
Bonus
If a user searches for -well-known huge ports MySQL treats that as not include *well*, could include *known* and *huge*. Of course you could solve that with an other extra query variant as well, but with the trick above you remove the hyphen so the search query looks simply like that:
SELECT id
FROM texts
WHERE MATCH(text) AGAINST('-wellknown huge ports' IN BOOLEAN MODE)
I have keywords like "some-or-other" where the hyphens matter in the search through my mysql database. I'm currently using the fulltext function.
Is there a way to escape the hyphen character?
I know that one option is to comment out #define HYPHEN_IS_DELIM in the myisam/ftdefs.h file, but unfortunately my host does not allow this. Is there another option out there?
Here's the code I have right now:
$search_input = $_GET['search_input'];
$keyword_safe = mysql_real_escape_string($search_input);
$keyword_safe_fix = "*'\"" . $keyword_safe . "\"'*";
$sql = "
SELECT *,
MATCH(coln1, coln2, coln3) AGAINST('$keyword_safe_fix') AS score
FROM table_name
WHERE MATCH(coln1, coln2, coln3) AGAINST('$keyword_safe_fix')
ORDER BY score DESC
";
From here http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html
One solution to find a word with a dashes or hyphens in is to use FULL TEXT SEARCH IN BOOLEAN MODE, and to enclose the word with the hyphen / dash in double quotes.
Or from here http://bugs.mysql.com/bug.php?id=2095
There is another workaround. It was recently added to the manual:
"
Modify a character set file: This requires no recompilation. The true_word_char() macro
uses a “character type” table to distinguish letters and numbers from other
characters. . You can edit the contents in one of the character set XML
files to specify that '-' is a “letter.” Then use the given character set for your
FULLTEXT indexes.
"
Have not tried it on my own.
Edit: Here is some more additional info from here http://dev.mysql.com/doc/refman/5.0/en/fulltext-boolean.html
A phrase that is enclosed within double quote (“"”) characters matches only rows that contain the phrase literally, as it was typed. The full-text engine splits the phrase into words and performs a search in the FULLTEXT index for the words. Prior to MySQL 5.0.3, the engine then performed a substring search for the phrase in the records that were found, so the match must include nonword characters in the phrase. As of MySQL 5.0.3, nonword characters need not be matched exactly: Phrase searching requires only that matches contain exactly the same words as the phrase and in the same order. For example, "test phrase" matches "test, phrase" in MySQL 5.0.3, but not before.
If the phrase contains no words that are in the index, the result is empty. For example, if all words are either stopwords or shorter than the minimum length of indexed words, the result is empty.
Some people would suggest to use the following query:
SELECT id
FROM texts
WHERE MATCH(text) AGAINST('well-known' IN BOOLEAN MODE)
HAVING text LIKE '%well-known%';
But by that you need many variants depending on the used fulltext operators. Task: Realize a query like +well-known +(>35-hour <39-hour) working week*. Too complex!
And do not forget the default len of ft_min_word_len so a search for up-to-date returns only date in your results.
Trick
Because of that I prefer a trick so constructions with HAVING etc aren't needed at all:
Instead of adding the following text to your database table: "The Up-to-Date Sorcerer" is a well-known science fiction short story. copy the hyphen words without hypens to the end of the text inside a comment: "The Up-to-Date Sorcerer" is a well-known science fiction short story.<!-- UptoDate wellknown -->
If the users searches for up-to-date remove the hyphen in the sql query:
MATCH(text) AGAINST('uptodate ' IN BOOLEAN MODE)
By that you're user can find up-to-date as one word instead of getting all results that contain only date (because ft_min_word_len kills up and to).
Of course before you echo the texts you should remove the <!-- ... --> comments.
Advantages
the query is simpler
the user is able to use all fulltext operators as usual
the query is faster.
If a user searches for -well-known +science MySQL treats that as not include *well*, could include *known* and must include *science*. This isn't what the user expected. The trick solves that, too (as the sql query searches for -wellknown +science)
Maybe simpler to use the Binary operator.
SELECT *
FROM your_table_name
WHERE BINARY your_column = BINARY "Foo-Bar%AFK+LOL"
http://dev.mysql.com/doc/refman/5.0/en/cast-functions.html#operator_binary
The BINARY operator casts the string following it to a binary string. This is an easy way to force a column comparison to be done byte by byte rather than character by character. This causes the comparison to be case sensitive even if the column is not defined as BINARY or BLOB. BINARY also causes trailing spaces to be significant.
My preferred solution to this is to remove the hyphen from the search term and from the data being searched. I keep two columns in my full-text table - search and return. search contains sanitised data with various characters removed, and is what the users' search terms are compared to, after my code has sanitised those as well.
Then I display the return column.
It does mean I have two copies of the data in my database, but for me that trade-off is well worth it. My FT table is only ~500k rows, so it's not a big deal in my use case.
Having problems getting the following query to work. I want to match the actual string " to control word relevance.
SELECT * FROM (table)
WHERE MATCH (field) AGAINST ("+<foo><![CDATA[1850]" IN BOOLEAN MODE)
When I run this it returns almost all records in the database, not just those which match the exact string.
AFAIK you can not use special characters in full text search indexes. It is limited to TEXT. (Words to be exact. For example you can have a list of most common words to be excepted form this index). You have to use LIKE if you are searching for pieces of code with special characters.