I have an issue with a full text search in a MySQL table.
MATCH AGAINST returns no results even I if have 7 records containing the words I'm looking for. What can I do to make it return the rows?
SELECT *
FROM site_plugin_products_cache_texts
WHERE MATCH(item_text) AGAINST ('+your +name' IN BOOLEAN MODE);
No rows.
SELECT *
FROM site_plugin_products_cache_texts
WHERE item_text LIKE'%your name%'
7 rows (0.071 s)
http://sqlfiddle.com/#!9/70c3fa/1/0
Thanks.
Both of those words are included in the Stopwords for MyISAM Search Indexes, which is essentially a list of words to completely ignore in fulltext indexing (as typically they tend to occur too frequently).
The simplest solution would be to switch to InnoDB as your engine, as it has a much shorter list of stopwords. If you need to use MyISAM, you will have to do the following:
To override the default stopword list for MyISAM tables, set the ft_stopword_file system variable. (See Section 5.1.7, “Server System Variables”.) The variable value should be the path name of the file containing the stopword list, or the empty string to disable stopword filtering. The server looks for the file in the data directory unless an absolute path name is given to specify a different directory. After changing the value of this variable or the contents of the stopword file, restart the server and rebuild your FULLTEXT indexes.
Related
I have a database of articles that I want to search through. I had been using normal Django ORM to search, which was getting way to slow and then I got to know a little about Indexes in Django. I'm using MySQL and I now know that with MYSQL I cannot put an index field into a TextField as described here in this stack question which I was facing. However in my case I can't change this to CharField.
I was reading through the MyQSL Docs which stated
MySQL cannot index LONGTEXT columns specified without a prefix length
on the key part, and prefix lengths are not permitted in functional
key parts.
Hence I was of the understanding that since TextField in Django is LONGTEXT for MYSQL, I came across this Django-MySQL package here and thought that using this if I could change the LONGTEXT to a MEDIUMTEXT using this package, this might get resolved. So my updated model I did this
class MyModel(Model):
........
document = SizedTextField(size_class=3)
However, I still see the same error while applying python manage.py makemigrations
django.db.utils.OperationalError: (1170, "BLOB/TEXT column 'document'
used in key specification without a key length")
How can I go about resolving this?
returning all the articles that contain a given word passed by the client. So would be something SELECT * from articles WHERE text CONTAINS searchword
Add
FULLTEXT(text)
and use
WHERE MATCH(text) AGAINST("searchword")
or perhaps
WHERE MATCH(text) AGAINST("+searchword" IN BOOLEAN MODE)
It will run very fast. There are caveats -- short words and "stop" words (like "the") are ignored.
(If DJango cannot facilitate that, then you have to do it with "raw SQL".)
All of these related types, TEXT, MEDIUMTEXT, and LONGTEXT, are too large to be indexed without specifying a prefix. An index prefix means that only the first N characters of the string are included in the index. Like this:
create table mytable (
t text,
index myidx (t(200))
);
The prefix length in this example is 200 characters. So only the first 200 characters are included in the index. Usually this is enough to help performance, unless you had a large number of strings that are identical in their first 200 characters.
The longest prefix that MySQL supports depends on the storage engine and the row format. Old versions of MySQL support index prefix up to 768 bytes, which means a lesser number of characters depending on if you use multi-byte character sets like utf8 or utf8mb4. The recent versions of MySQL default to a more modern row format, which supports up to 3072 bytes for an index, again reduced by 3 or 4 bytes per character.
I'm not a regular Django user, so I tried to skim the documentation about defining indexes on model classes. But given a few seconds of reading, I don't see an option to declare a prefix for an index on a long string column.
I think your options are one of the following:
Change the column to a shorter string column that can be indexed
Create the index using the MySQL client, not using Django migrations
I'm using MySQL FullText Index Search on my system and when I search with 3 characters the query works well but when I search with 2 characters the query returns null.
I use this query:
select *
from table
where MATCH (description) AGAINST ('+gp*' IN BOOLEAN MODE)
I already use this command show variables like 'ft%' to get the value of ft_min_word_len and the value is 4. It is strange that with 3 characters the query works.
However i tried to change this value on file /root/.my.cnf. (I use Ubuntu 14.04 and server pilot). I add this to the file:
[mysqld]
innodb_ft_min_token_size=2
ft_min_word_len=2
Restart the mysql, drop index and create index.
Although when I execute this command again show variables like 'ft%' the value still remains 4.
How do I use mysql full text search with 2 characters?
Any help will be appreciated!
Correct location of the my.cnf?
/etc/mysql/my.cnf
Take a look here or here
I am facing issue with MySQL, UNCOMPRESS function.
I have table named as user and user_details stores COMPRESS values. In this case before search for values from the user_details i have to UNCOMPRESS it.
But issue is after I do UNCOMPRESS, search become case-sensetive.
Like.. e.g
If I tries below sql, it will only search for values which contain CAPITAL TESTING word and ignore small case testing word
SELECT * FORM user WHERE UNCOMPRESS(user_details) LIKE '%TESTING%'.
I want case-insensitive search.
But issue is after I do UNCOMPRESS, search become case-sensetive.
This is because COMPRESS() "Compresses a string and returns the result as a binary string." (emphasis mine)
When you perform a LIKE operation on a binary string, a binary comparison will be performed (which is case-sensitive).
You may be able to circumvent this by putting a CAST() around the COMPRESS() statement.
But you probably shouldn't be doing this in the first place. It's an extremely inefficient way to search through huge amounts of data. MySQL will have to uncompress every row for this operation, and has no chance of using any of its internal optimization methods like indexes.
Don't use COMPRESS() in the first place.
Try this:
SELECT * FROM `user` WHERE LOWER(UNCOMPRESS(user_details)) LIKE '%testing%'
But as Pekka well pointed or its very inefficient . If your using MyIsam Engine another alternative is myisampack which compresses the hole table and its still query-able.
I am using fulltext searching in mysql to search a database of videos I have, however when I search my videos some results will never get returned because the title I am searching for is less than the ft_min_word_len set in MySQL's settings.
mysql_query("SELECT MATCH(videoDescription) AGAINST('".$searchString."' IN NATURAL LANGUAGE MODE)
FROM videos
LIMIT ".$start.",".$end."");
I tried firing up the mysql console to change the setting, however it told me it is read only. The only possible solution I have seen is to have the setting overridden at startup with option files. How do I use option files to overwrite the ft_min_word_len to 2 at startup?
ft_min_word_len is a system variable, that has to be set at the startup of the MySQL server.
This can be done passing parameters on the command-line used to start MySQL, or (recommended, I'd say), using a file containing options -- generally, for example, something like /etc/my.cnf or /etc/mysql/my.cnf should do the trick.
For more informations about setting system variables, you can take a look at the following section of the manual :
4.2.3. Specifying Program Options
Also, don't forget that you'll have to rebuild your fulltext index, after changing that parameter, so the new value is taken into account.
I have written a query which will perform Full Text search using full search Index in mysql Table.
But my problem is that when user searches with "to go" then it will not search anything because of stopwords in mysql.
So my question is, how can I write a Full Search query which will ignore the stopwords?
To override the default stopword list, set the ft_stopword_file system variable. (See Section 5.1.4, “Server System Variables”.) The variable value should be the path name of the file containing the stopword list, or the empty string to disable stopword filtering. The server looks for the file in the data directory unless an absolute path name is given to specify a different directory. After changing the value of this variable or the contents of the stopword file, restart the server and rebuild your FULLTEXT indexes.
http://dev.mysql.com/doc/refman/5.1/en/fulltext-fine-tuning.html