Text search in MySQL - Performance and Alternatives - mysql

Question

I have a set of tables in MySQL like this (foreign keys referenced by [table_name]_id):
Articles(id, author_id, title, date, broad_search, ...)
Keywords(id, article_id, keyword (varchar))
Authors(id, name, ...)
Attachments(id, article_id, url, ...)
The table we are concerned about most is 'Keywords' so I am mentioning the indexes only for it:
id - Primary - BTREE
(article_id,keyword) - Unique - BTREE
keyword - BTREE
article_id - BTREE
Each Article has associated list of Keywords. The "broad_search" column in Articles states whether that particular article can be matched broadly with the keywords (broad_search=1) or if it has to be an exact match of the keyword(broad_search=0). I have a SELECT query which pulls a list of articles based on keywords, broad_search parameter and other filter criteria.
$sql = "SELECT *
FROM Keywords k, Attachments at, Articles ar, Authors a (2 more tables)
WHERE
((ar.broad_search=0 AND k.keyword = '$Keyword')
OR (ar.broad_search=1 AND (INSTR('$Keyword', k.keyword)>0 OR k.keyword like '%$Keyword%')))
AND at.article_id = ar.id
AND a.id = ar.author_id
... (more conditions)
LIMIT 20";
An article can be set to either braod match or exact match, and I'm trying to get a list of them based on a keyword.
Exact match is straightforward. But broad match has various cases which will not let me use a simple wild card pattern like '%search_term%'. An example:
Keywords for a broad match article = {books, used books, reading books, popular book}
search term = new books
Now, we cannot use the mysql wildcard string matching as '%new books%' will not match any of the keywords but it needs to be retrieved as the search term contains a substring of the keywords (broad_search=1). So, broad_search is of 2 types: search_term = "cars" in keyword "used cars" and search term = "used cars" in keyword "cars".
If broad_search=0, do an exact match. If broad_search=1, match both cases:
((ar.broad_search=0 AND k.keyword = '$Keyword')
OR (ar.broad_search=1 AND (INSTR('$Keyword', k.keyword)>0 OR k.keyword like '%$Keyword%')))
The query I wrote perfectly does the job. But the issue is with performance. The keywords table is very large with 100,000+ rows and keeps growing. Also, this is a high load app and kills my server due to the huge number of requests it receives.
I feel this is not the right way to perform a text search. I tried going through mysql docs regarding full text search but I did not quite understand it's application and if it fits my search criteria. Also, I was thinking if Apache Lucene would be a better choice, but I haven't used it earlier so not really sure (this query runs in a PHP script).
How should I be implementing this? Is it indexing issue, or is the MySQL INSTR function inefficient, or should I be using a whole different approach?

Answer 1

MySQL isn't a search engine, it's a Relation Database Management System (RDBMS). However, you can implement native MySQL tools to emulate Full-Text searching capabilities, such as setting up a search table as MyISAM and adding a FULLTEXT index to columns you wish to search upon. You can read the MySQL docs for more info on how MySQL supports Full-Text searching.
Even if you get Full-Text search queries to work the way you want, you will still miss out on a whole host of features that a true search engine (Lucene) supports. Features such as a facets, spatial searches, result boosting, weighting, etc. I'd suggest you read up on Apache SOLR, as it supports all these features and many more. There is even a PHP SOLR API which you can use to access a SOLR instance.
I'm not saying to abandon MySQL altogether, but use it for it's intended purpose, to persistently store data which can be queried upon, and which can be uses to populate your search engine indices. SOLR even has a built in Document Import Handler, which will allow you to set a database query to be used when you want to mass import data from your MySQL database.
The learning curve is relatively high, as it is with learning most new technologies, but when you are done you will wonder how you ever got by without using a true Full-Text search engine.