Have a table with ~25K rows, the user can provide keywords and negative keywords to filter rows.
It slows down when a user added a lot of keywords and/or negative keywords.
The query looks like this:
SELECT id, title, description
FROM entities
WHERE
(
title LIKE '%keyword_1%' OR description LIKE '%keyword_1%'
OR title LIKE '%keyword_2%' OR description LIKE '%keyword_2%'
OR title LIKE '%keyword_3%' OR description LIKE '%keyword_3%'
)
AND
(
title NOT LIKE '%negative_keyword_1%' OR description NOT LIKE '%negative_keyword_1%'
OR title NOT LIKE '%negative_keyword_2%' OR description NOT LIKE '%negative_keyword_2%'
OR title NOT LIKE '%negative_keyword_3%' OR description NOT LIKE '%negative_keyword_3%'
)
for example, a query with 9 keywords and 130 negative keywords takes ~7 seconds.
Maybe there is a better solution to filter those rows without LIKE? maybe the whole logic is wrong.
Tried MATCH () AGAINST() - it is slower than LIKE for some reason.
For such cases like yours it might help to try to filter the positive matches in database and then test them for a presence of a negative keyword in memory. So try something like:
SELECT id, title, description
FROM entities
WHERE
(
title LIKE '%keyword_1%' OR description LIKE '%keyword_1%'
OR title LIKE '%keyword_2%' OR description LIKE '%keyword_2%'
OR title LIKE '%keyword_3%' OR description LIKE '%keyword_3%'
)
This gives you results which match positive keywords. You can then filter out the matches in memory which contain negative keywords.
Also give an index a chance to be used with LIKE. This reduces your search options from CONTAINS to STARTS WITH but it might be sufficient for your case. Syntax is LIKE 'keyword_1%'.
Fulltext search
Another option is to use MySQL fulltext search by defining a fulltext index on your title and description columns.
CREATE FULLTEXT INDEX idx_title ON entities(title);
CREATE FULLTEXT INDEX idx_description ON entities(description);
Or you can merge these two columns into a single column - for the search purposes. Then you need only one fulltext index.
Query syntax is then following:
MATCH (title) AGAINST ('keyword_1')
instead of
title LIKE '%keyword_1%'
For this search I would also recommend to filter positive matches only in a database and then filter out the matches in memory which contain negative keywords.
For years when I want my user to search some field in my database where he can type anything he wants I use an algorithm to break the words and search each word separetely... a mess.
For example, if the user types in the search box "aaa bbb ccc" I dont like using:
SELECT id
FROM table
WHERE description LIKE '%aaa bbb ccc%'
Cause sometimes the user types things out of order and the query above wouldng find. What I usually do is breaking the string and concatenating it with PHP so the result becomes:
SELECT id
FROM table
WHERE description LIKE '%aaa%'
AND description LIKE '%bbb%'
AND description LIKE '%ccc%'
But today after talk to a friend I was wondering if there is some native way to do this faster using MY SQL?
What you want to do is called full text search and most relational databases support it nowadays, including mysql.
I think you can use REGEXP. For example:
Select * from table where description REGEXP 'aaa|bbb|ccc'
FULLTEXT Searches are really fast.
INSTR or locate works better than REGEXP. But it depends on various factors.
More comparison here
SELECT * from table where INSTR(description, 'aaa') >0
SELECT * from table where LOCATE(description, 'aaa') >0
My concept is little unclear. For my site I want to create a searchbox. And I will use Autocomplete function to check related keywords from database. So while typing matching word will come as suggestion. But I am not sure about how to check it through multiple tables. My concept is on health based site so there is different table for each sections like hospitals, doctors, laboratories, chemist etc. I just need guidance how should I do it?
You can use union queries to return autocomplete data for your textbox. For example, you could write a query that for a search term could do this:
select hospitals.name as name
from hospitals
where hospitals.name like '%?%'
UNION
select doctors.lastname as name
from doctors
where doctors.lastname like '%?%'
UNION
select laboratories.labname as name
from laboratories
where laboratories.labname like '%?%'
UNION
...
Be careful though: the performance of this could degrade fast, especially for "contains" searches like the example above ( like '%?%').
A faster query would be a "starts with" which would change the like clause in the query above to like '?%'
So I have a DB and a web page where I want to display the result of my db search
I have try a couple ways both work but they are incomplete for what I want to accomplish
I want to be able to search my table columns id,Id_Nombre,Pais,Estado,Ciudad,website all of them with a word or several words from my search text.
this code works but I have to type exactly the word it's case sensitive:
$query = "SELECT * FROM Medios_table WHERE concat(id,Id_Nombre,Pais,Estado,Ciudad,website) LIKE '%$Busqueda%'";
so as a result i I type a word like "People" in my search box and in my data base that word is type in any of those columns as "people" it wont find it.
second code I use works but only using one column
$query = "SELECT * FROM Medios_table WHERE UPPER(Id_Nombre) LIKE UPPER('%$Busqueda%')";
the result for this code is great since it will find it no matter the case used, but I need to extend this type of search to all the other columns, but so far everithing I use does not work.
I have tried:
$query = "SELECT * FROM Medios_table WHERE UPPER(id,Id_Nombre,Pais,Estado,Ciudad,website) LIKE UPPER('%$Busqueda%')";
$query = "SELECT * FROM Medios_table WHERE UPPER(concat(id,Id_Nombre,Pais,Estado,Ciudad,website)) LIKE UPPER('%$Busqueda%')";
$query = "SELECT * FROM Medios_table WHERE concat(UPPER(id,Id_Nombre,Pais,Estado,Ciudad,website)) LIKE UPPER('%$Busqueda%')";
etc.
any help is greatly appreciated, thanks.
Did you try UPPER around each column name? Like this:
$query = "SELECT * FROM Medios_table WHERE concat(UPPER(id),UPPER(Id_Nombre),UPPER(Pais),UPPER(Estado),UPPER(Ciudad),UPPER(website)) LIKE UPPER('%$Busqueda%')";
Your upper(concat(...)) version should work. If you had a problem with that, it's probably just a typo, like you left out a parenthesis or something.
You can't say upper(x,y,z) because upper takes only one parameter. You must do the concat first, then do the upper, i.e. upper(concat(x,y,z)).
That said, this query will be very slow on a big database, because the db engine has to read every record in the table, and then for each one search it character by character. If the table is small or this is done infrequently, that might be acceptable. If the table is big and/or this query will be executed often, you really need a totally different approach.
Update
If you really need to search against any text in a field, where you cannot make any assumptions about the text being searched in or the text being searched for in advance, you should investigate fulltext searches. http://dev.mysql.com/doc/refman/5.0/en/fulltext-natural-language.html
If you will be doing these sort of searches all the time, you might want to build a dictionary, that is, build a list of all the word with all the records that that word occurs in. I think that's basically what fulltext does, so this may or may not gain you anything.
But if you have some foreknowledge, that is, if it's not really that you want to search for arbitrary text occurring anywhere in arbitrary text, than break out the things you want to search for into separate fields.
To take a simple example, suppose you have a field that contains customer full name, like "Fred Smith", "Mary Jones", etc. You want to search for someone by last name. You could search for
where full_name like '%Smith%'
But this would require reading every record in the table. If, instead, you broke the field into first name and last name, then you could search for
where last_name='Smith'
If you have an index on last_name this would be a very fast search.
If you're just trying to look for an entered value in any of several fields, it is much, much faster to do
where estado='Toledo' or ciudad='Toledo' or pais='Toledo'
(assuming that you have indexes on estado, ciudad, and pais), then
where concat(estado, ciudad, pais) like '%Toledo%'
(where no index will do you any good).
If you want to do case-insensitive searches, create an index on upper(estado) instead of on estado, etc. Then "where upper(estado)='TOLEDO'" can use the index.
Also, I'm not sure about MySql, but some database engines can use an index if the LIKE does not begin with a wildcard. That is, "somefield like '%x%'" must read every record in the table. But on some db's, "somefield like 'x%'" can use can index to get to the records where the field starts with "x" and then just process those.
You need full-text search
This link may be useful Mysql full-text search
I have a page that gets all rows from a table in a database, then displays the rows in an HTML table.
That works great, but now I want to implement a 'search' feature. There is a searchbox, and search-terms are separated by a space. I am going to make it search three fields for the search terms, 'make' 'model' and 'type.' These three fields are VARCHAR(30).
Currently if I wanted to search using 3 terms (say 'cool' 'abc' and '123') my query would look something like this.
SELECT * FROM table WHERE make LIKE '%cool%' OR make LIKE '%abc%' OR make LIKE '%123%' OR model LIKE '%cool%' OR model LIKE '%abc%' OR model LIKE '%123%' OR type LIKE '%cool%' OR type LIKE '%abc%' OR type LIKE '%123%'
That looks really bad, and it will get even worse if there are more search terms or more fields to search.
My question to you: is there a better way to search? If so, what?
Use REGEXP instead of like
SELECT * FROM table WHERE make REGEXP 'cool|abc|123' OR model REGEXP 'cool|abc|123' OR type REGEXP 'cool|abc|123';
read more about REGEXP
http://dev.mysql.com/doc/refman/5.1/en/regexp.html
http://www.go4expert.com/forums/showthread.php?t=2337
You should use FULLTEXT. Search SO about it, it is very easy and useful.
You could use a JQuery Table Plugin, like data tables Features on-the-fly filtering which you could use instead of a lengthy sql command. And its free.