I am currently looking into using FULLTEXT indexes in MySQL for search functionality within a web site.
Basically, the user can go to an advanced search page, and select 1 or more columns to search against, e.g. they can search Title, Description and Comments or either only 1 column or a mixture of the three and when they perform the search these selected columns are searched for against the keywords.
I had created 1 index for the title, 1 index for the description and 1 index for the comments and then tried to run the following query:
SELECT * FROM support_calls WHERE MATCH(Title, Description) AGAINST('+these, +are, +some, +keywords')
I got an error from MySQL saying that the MATCH didn't match any fulltext indexes and I found that I need to create an index which included Title and Description together instead of having them in separate indexes.
This is going to add some complexity if this is the case as I am going to have to create an index for every single variation of what columns the user selects. Am I going about this the right away or is there a better solution?
first execute below query and then run your MATCH() query.
ALTER TABLE support_calls ADD FULLTEXT (
Title, Description
)
Related
I was trying to make a very fast & efficient approach to fetch the records using keywords as search.
Our MYSQL table MASTER tablescontains 30,000 rows and has 4 fields.
ID
title (FULLTEXT)
short_descr (FULLTEXT)
long_descr (FULLTEXT)
Can any one suggest which is one more efficient?
LIKE %
MYSQL's AGAINST
It would be nice if some one can write a SQL query for the keywords
Weight Loss Secrets
SELECT id FROM MASTER
WHERE (title LIKE '%Weight Loss Secrets%' OR
short_descr LIKE '%Weight Loss Secrets%' OR
long_descr LIKE '%Weight Loss Secrets%')
Thanks in advance
The FULLTEXT index should be faster, maybe its a good idea to add all columns into 1 fulltext index.
ALTER TABLE MASTER
ADD FULLTEXT INDEX `FullTextSearch`
(`title` ASC, `short_descr` ASC, `long_descr` ASC);
Then execute using IN BOOLEAN MODE
SELECT id FROM MASTER WHERE
MATCH (title, short_descr, long_descr)
AGAINST ('+Weight +Loss +Secrets' IN BOOLEAN MODE);
This will find rows that contains all 3 keywords.
However, this wont give you exact match the keywords just need to be present in same row.
If you also want exact match you could do like this, but its a bit hacky and would only work if your table doesnt get to big.
SELECT id FROM
(
SELECT CONCAT(title,' ',short_descr,' ', long_descr) AS SearchField
FROM MASTER WHERE
MATCH (title, short_descr, long_descr)
AGAINST ('+Weight +Loss +Secrets' IN BOOLEAN MODE)
) result WHERE SearchField LIKE '%Weight Loss Secrets%'
i'm not very very experimented with the indexes so that's why i'm asking this silly question. i searched like everywhere but i didn't get a clear answer.
I will have a table items with columns: id,name,category,price
Here will be 3 indexes:
id - Primary Index
name - FullText Index
category,price - Composite Index
I estimate my table in future will get like 700.000-1.000.00 rows.
I need to do a fulltext search for name and where category is a specified category and order by price.
So my query will be this:
SELECT * FROM items
WHERE MATCH(name) AGAINST(‘my search’) and category='my category' order by price
My question is:
How many index will be used to perform this search?
It will use 2 indexes?
[fulltext index] & [category,price] index - Will get results for words and then will use the next index to match my category and price order
It will use 1 index
[fulltext index] only - Will get results for words, but after will have to manually match my category and price order
I want my query to be fast, what are you opinions? I know the fulltext search is fast, but what happen if i apply clauses like: category and price order? will be same fast?
MySQL will only ever use one index in any search. The reason being that using two indexes will require two searches. This will make the query much more slower. You can force MySQL to use a specific index in a query but this is not a good idea.
In summary: MySQL will only ever use one index it cant use two indexes.
I wanted to build my first real search function. I've been Googling a while, but wasn't able to really make my mind up and understand everything.
My database exists of three InnoDB tables:
Products: Contains of a product information. Columns: proID (primary, auto-increment), content (contains up to a few hundred words), title, author, year, and a bunch of others that are not related to the search query. Rows: 100 to 2000.
Categories: Contains category information: Columns: catID (primary, auto-increment), catName. Rows: 5-30
Productscategories: Link between the two above. Each product can be related to multiple categories. Columns: pcID (primary, auto-increment), catID, proID. Rows: 1-5 times amount of products.
My search function offers the following things. They do not have to be filled in. If more than one is filled in, the final query will connect them with the AND-query:
Terms: Searches the content and title field. Searches on random terms, multiple words can be added, but searches for each of them seperate. Most likely 1 match with the database should be enough for a hit (OR-query)
Year: Searches on the year-column of products.
Category: Selectable from a list of categories. Multiple possible. The form returns the catID's of the chosen categories. 1 match with the database should be enough for a hit (OR-query)
Author: Searches on the author-column of products
As you may have noticed, when a category is selected, the tables products and productcategories are joined together for the search query. There is also a foreign key set between the two.
To clearify the relations an example of how it should be interpreted(no search for the year!):
Search WHERE (products.content = term 1 OR products.content = term 2 OR products.title = term 1 OR products.title = term 2 ......) AND (products.author = author) AND (productscategories.catID = catID1 OR productscategories.catID= catID2 ......)
Also note that I created a pagination system that only shows 10 results on each 'page'.
The question I am stuck with is the following: I wish to optimize this search query, but can't figure out which way is the best.
Most cases I found Googling used the LIKE %% mysqli-query. However some used the MATCH...AGAINST. I seem to really like the last one because I read it can sort on relevance and because it seems to make the query a lot easier to create (1 match against the term values instead of plenty of LIKE %% combined with OR). It seems I would only use it on the Term-searchfield though. However for MATCH...AGAINST I will need a MyIsam table (right?), in which I can't use the foreign key to prevent faults in the database.
MATCH...AGAINST example (without year field, category field and not joining products and productscategories):
SELECT *,MATCH (content,title) AGAINST ('search terms' IN BOOLEAN MODE) AS relevance
FROM products WHERE (MATCH (content,title) AGAINST ('search terms' IN BOOLEAN MODE)) AND
author='author' SORT BY relevance LIMIT 10
%LIKE% example(without year field, category field and not joining products and productscategories) and sadly no relevance sorting:
SELECT * FROM products WHERE
(content LIKE '%term1%' OR content LIKE '%term2' OR title LIKE '%term1%' OR title LIKE '%term2%')
AND (author='author') SORT BY title LIMIT 10
I could make a relevance sorting by using the CASE and add 'points' if a term comes in the title or the content? Or would that make the query too heavy for performance?
So what is the best way to make this kind of query? Go with Innodb and LIKE, or switch to MyIsam and use the MATCH...AGAINST for sorting?
You dont have to switch to MyIsam. Fulltext indexing is supported in Mysql 5.6 and higher.
I usually recommend using fulltext indexes. Create a fulltext index on your columns title,author,year
Then you can run a fulltext query on all 3 at the same time, and apply IN BOOLEAN MODE to really narrow your searches. This is ofcourse something you have to decide for yourself but the options in fulltext are more.
However, if you are running queries that spawn between a range, date for instance or a simple string. Then a standard index is better but for tekst searching in different columns, fulltext index is the way to go!
Read this: http://dev.mysql.com/doc/refman/5.6/en/fulltext-search.html
I have a database with 75,000+ rows with 500+ entries added per day.
Each row has a title and description.
I created an RSS feed which gives you the latest entries for a specific search term (ex. http://site.com/rss.rss?q=Pizza would output an RSS for the search term "Pizza").
I was wondering what would be the best way to write the SQL query for this. Right now I have:
SELECT *
FROM 'table'
WHERE (('title' LIKE %searcherm%) OR ('description' LIKE %searcherm%))
LIMIT 20;
But the problem is it takes between 2 to 10 seconds to execute the query.
Is there a better way to write the query, do I have to cache the results (and how would I do that?) or would changing something in the database structure speed up the query (indexes?)
A relatively simple solution for this would be incorporating a FULLTEXT index on these two fields and subsequently searching by using this index.
ALTER TABLE table ADD FULLTEXT(title, description);
Then would you need to perform a search, you'd do the following:
SELECT id FROM table
WHERE MATCH (title, description) AGAINST ('keyterm');
Fulltext indexed search is the automatic solution included in most SQL databases. It's much speedier comparing to doing LIKES. This is also optimized for your specific case because you are only interested in natural language search terms.
As well, fulltext index has some limiting algorithm for detecting relevancy. You can read more about it here
EDIT
In the alter statement, I missed the fulltext index name, it should be:
ALTER TABLE table ADD FULLTEXT ft_index_name(title, description);
Try:
SELECT * FROM table
WHERE MATCH (title,description) AGAINST (searchterm);
Make sure you add a full text index on title, description together.
Dont try to reinvent the wheel. MATCH and AGAINST are provided by mysql to do exactly that and to make your life easy. However, note full text search works on MyISAM tables. You can workaround for InnoDb too. You can simply add FT index by altering table like:
ALTER TABLE table ADD FULLTEXT(title,description);
If you're using a query with LIKE '%term%' the indexes can't be used. They can be used only if you use a query like 'term%'. Think about an address book with tabs, you can find really fast contacts starting with letter L, but to find contacts with a on somewhere in the word, you've to scan the whole addressbook.
The better alternative could be to use full text indexes:
CREATE FULLTEXT INDEX title_desc
ON table (title, description)
And then in the query:
SELECT title, description FROM table
WHERE MATCH (title, description) AGAINST ('+Pizza')
I would go with JohnB's or gtr32x's answer (Full Text Indexing). To complement their answer, there's a manual way to create a simple full text index that's simple and it's super fast...
Split title and description into keywords, and place them in a Keywords table, which has a foreign key to the original RSS article. Make sure the keyword column in Keywords is indexed. The you can do something like:
SELECT DISTINCT ra.*
FROM RssArticle ra
INNER JOIN Keywords k ON k.ArticleID = ra.ArticleID
WHERE k IN ( 'SearchTerm1', 'SearchTerm2', 'SearchTerm3')
LIMIT 20;
And it's fast!
Try either of the following four queries:
select * from myTable where concat_ws(' ',title,description) like '%pizza%';
select * from myTable where concat_ws(' ',title,description) regexp '.*pizza+.*';
select title,description from myTable where concat_ws(' ',title,description) like '%pizza%';
select title,description from myTable where concat_ws(' ',title,description) regexp '.*pizza+.*';
the point is to use concat before searching
A few pointers: Drop the * in your select statement and pull only the searched criteria, and make sure to add indexes to the columns that are getting searched.
SELECT `title`,`description`
FROM `table`
WHERE `title` LIKE '%$searchterm%' OR `description` LIKE '%$searchterm%' LIMIT 25;
Did you create an index for title and for description?
You should consider Sphinx for Full Text Search capabilities.
Thanks for the comment Tyler.
I restate my answer:
1) Create an index on title and description columns, but your query would be limited to the example below, and that's not ideal for finding all relevant rows:
SELECT *
FROM 'table'
WHERE title LIKE 'searcherm%' OR description LIKE 'searcherm%'
LIMIT 20;
2) As others have mentioned, use MySQL Full-Text Search, but you are limited to MyISAM table engine, as it isn't available for InnoDB. However, you can mix engines in MySQL, so you can make this table MyISAM, even if all your other tables are InnoDB.
3) Use an external Full-Text Search engine, such as Sphinx. This will give you more relevant search results (MySQL Full-Text Search leaves much to be desired), it will perform better, and it abstracts the burden of Full-Text Searching away from your database.
I'm deploying a Rails application that aggregates coupon data from various third-party providers into a searchable database. Searches are conducted across four fields for each coupon: headline, coupon code, description, and expiration date.
Because some of these third-party providers do a rather bad job of keeping their data sorted, and because I don't want duplicate coupons to creep into my database, I've implemented a unique compound index across those four columns. That prevents the same coupon from being inserted into my database more than once.
Given that I'm searching against these columns (via simple WHERE column LIKE %whatever% matching for the time being), I want these columns to each individually benefit from the speed gains to be had by indexing them.
So here's my question: will the compound index across all columns provide the same searching speed gains as if I had applied an individual index to each column? Or will it only guarantee uniqueness among the rows?
Complicating the matter somewhat is that I'm developing in Rails, so my question pertains both to SQLite3 and MySQL (and whatever we might port over to in the future), rather than one specific RDBMS.
My guess is that the indexes will speed up searching across individual columns, but I really don't have enough "under the hood" database expertise to feel confident in that judgement.
Thanks for lending your expertise.
will the compound index across all
columns provide the same searching
speed gains as if I had applied an
individual index to each column?
Nope. The order of the columns in the index is very important. Lets suppose you have an index like this: create unique index index_name on table_name (headline, coupon_code, description,expiration_date)
In this case these queries will use the index
select * from table_name where headline = 1
select * from table_name where headline = 1 and cupon_code = 2
and these queries wont use the unique index:
select * from table_name where coupon_code = 1
select * from table_name where description = 1 and cupon_code = 2
So the rule is something like this. When you have multiple fields indexed together, then you have to specify the first k field to be able to use the index.
So if you want to be able to search for any one of these fields then you should create on index on each of them separately (besides the combined unique index)
Also, be careful with the LIKE operator.
this will use index SELECT * FROM tbl_name WHERE key_col LIKE 'Patrick%';
and this will not SELECT * FROM tbl_name WHERE key_col LIKE '%Patrick%';
index usage http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html
multiple column index http://dev.mysql.com/doc/refman/5.0/en/multiple-column-indexes.html