I have this "log-queries-not-using-indexes" enabled in MySQL. I have one query which is being logged by MySQL as such i.e. query that is not using indexes. The thing is this query uses LIKE for e.g.
category like '%fashion%'
and if I remove LIKE or change it to
category = 'fashion'
then it says it is using indexes.
So when we are using LIKE in our query, MySQL will log it as not using indexes no matter what?
Thanks
Using a clause like %fashion% will never use a regular index. You need a full-text index if you want to do that kind of search.
Remember that a varchar indexes on first part of the string. So, if you are searching for an ocurrence of fashion on any part of the string, then index will offer no help to improve performance since you will need to search every single string.
However, if you only search for the first part like this:
select * from table where field like 'fashion%'
Then index could be used and could be helpful.
Related
I have a database with 2 tables, I must run a simple query `
select *
from tableA,tableB
where tableA.user = tableB.user
and tablea.email LIKE "%USER_INPUT%"
Where user_input is a part of the string of tablea.email that has to match.
The problem:
The table will carry about 10 million registers and its taking a while, the cache of ibatis (as far as I know) will be used only if the previous query looks the same. example: for USER_INPUT = john_doe if the second query is john_doe again the cache willt work, but if is john_do will not work(that is, as I said, as far as I know).
current, the tableA structure is like this:
id int(11) not_null auto_increment
email varchar(255)not_null
many more fields...
I dont know if email , a varchar of 255 might be too long and could take longer time because of that, if I decrease it to 150 characters for example, would the response time will be shorter?
Right now the query is taking too long... I know I could upgrade to more memory to the servers but I would like to know if there is other way to improve this code.
tableA and tableB have about 30 fields each and they are related by the ID on a relational schema.
Im going to create an index for tableA.email.
Ideas?
I'd recommend running an execution plan for that query in your DB. That'll tell how the DB plans to execute your query, and what you're looking for is something like a "full table scan". I'd guess you'll see just that, due to the like clause, and an index the email field won't help that part.
If you need to search by substrings of email addresses you might want to consider the granularity of how you store your data. For example, instead of storing email addresses in a single field as usual you could split them into two fields (or maybe more), where everything before the '#' is in one field and the domain name is in another. Then you could search by either component without needing a like and then indexes would significantly speed things up significantly. For example, you could do this to search:
WHERE tableA.email_username = 'USER_INPUT' OR tableA.email_domain = 'USER_INPUT'
Of course you then have to concatenate the two fields to recreate the email address, but I think iBatis will let you add a method to your data object to do that in a single place instead of all over your app (been a while since I used iBatis, though, so I could be wrong).
MySQL cannot utilize indexes on LIKE queries where the wildcard precedes the search string (%query).
You can try a Full-Text search instead. You'll have to add a FULLTEXT index to your email column:
ALTER TABLE tablea
ADD FULLTEXT(email);
From there you can revise your query
SELECT *
FROM tableA,tableB
WHERE tableA.user = tableB.user
AND MATCH (tablea.email) AGAINST ('+USER_INPUT' IN BOOLEAN MODE)
You'll have to make sure you can use full text indexes.
Full-text indexes can be used only with MyISAM tables. (In MySQL 5.6 and up, they can also be used with InnoDB tables.)
http://dev.mysql.com/doc/refman/5.5/en/fulltext-search.html
I found the following query in our MySQL slow query log:
SELECT target_status
FROM link_repository
WHERE target_url LIKE CONCAT('%', 'bundle/rpi_/activity/rpi_bridge/bridge_manual.pdf')
When I pointed this out to the developer manager in a conversation about slow page loads, he stated:
come on; concat() is a simple string concatenation and '%' is the wildcard in the search string. I know that searching strings is not the fastest of operations (that's why we have lucene-like engines, but this is trivial stuff)
There's about 18k rows in link_repository, which isn't much. The documentation I'm finding is that indexing on character strings doesn't work with wildcards. Is there an alternative strategy one can use?
In order for LIKE to use index it has to start with something. MySQL search from left to rigt. So if the string star with anything then MySQL will do a table scan and no index will work.
However, if you are using InnoDB tables you can try to use Full-Text Index.
You can add a Full-Text Index on the column, then you can use MATCH AGAINST function to find what you need then you can add RIGHT() clause to only give you the results that end with your string.
CREATE FULLTEXT INDEX target_url ON target_status(target_url);
Then you can query the records like so
SELECT target_status
FROM link_repository
WHERE MATCH ('bundle/rpi_/activity/rpi_bridge/bridge_manual.pdf') AGAINST(target_url) AND RIGHT(target_url, 49) = 'bundle/rpi_/activity/rpi_bridge/bridge_manual.pdf'
I am wondering how MySQL finds the rows in a table when searching like so:
select * from table where field = 'text';
Does it use a particular search algorithm? Is it practically the fastest way to look up information in a table? Or would building a search macro using another algorithm (like Boyer-Moore) work faster?
If there is an index on field, then databases often use a b-tree for indexed searches. If there is no index, then the entire table is scanned. This describes some of the techniques used in MySql
http://dev.mysql.com/doc/refman/5.5/en/index-btree-hash.html
Many hours of work has gone into optimizing MySql. Take advantage of that work already done, and resist trying to re-doing it
For that query it can do nothing other than searching every entry of that table and comparing its field column against that string.
Boyer-Moore isn't needed because it's exact equality that's requested and not asking whether the field contains that string.
If you are interested in how it found those records try executing using the EXPLAIN keyword:
EXPLAIN select * from table where field = 'text';
I would recommend looking at this article to get a better understanding what is happening in the background.
I would be very surprised if you would be able to write something on your own that is faster. You could look at creating indexes on the table in question to speed up selects.
I have a pretty simple LIKE search for MySQL that i'd like to transform into a fulltext. The problem is i need to be able to implement it so that it starts with X. Like the example below:
SELECT column FROM table WHERE column LIKE "startswith%"
as you can see that query returns all results that begins with "startswith". I need to do this with a fulltext.
Is this possible to do?
No, that isn't how fulltext works (it's actually just a list with loose words underneath, no information about location relative to the string) but there's no reason why you can't have that LIKE ... as an extra WHERE clause. FULLTEXT can still help to get a smaller subset of results if you haven't got another key on column. If you do have a key on column, using FULLTEXT for this is useless.
You can set a key on just the start of a column with ADD INDEX (column(123)); (which would only index the first 123 characters). THis also works for text/blob columns (in the latter case it's the binary length you give).
I am not sure about MySQL, but in SQL Server you can convert the column into a varchar and perform a LIKE on the result, such as below:
SELECT column FROM table WHERE CONVERT(varchar(255), column) LIKE 'startswidth%'
Since this is ANSI standard, I presume MySQL will handle this as well.
We are developing our database in MySql with innoDB engine. The database contains a column that is of varchar type with each entry containing about 3000 characters. We are to provide search on this column. For speeding up purpose, we need to add index on this column. Can you put in some information in this regard?
Which type of index do we need to put in to speed up the search? Do we need to take some other care about it for performance improvement?
If by search you mean you'll be performing a query like this:
SELECT * from cars WHERE car LIKE '%{search_str}%'
Then I am afraid that even if you add a key to the car column mysql will still have to perform a full-scan and your query might potentiolly be very slow.
If you are planning on supporting a significan amount of data to be searched and expect some high qps numbers, I would reccomend you have a look at Apaches Lucene project, which can drastically speed up any search query performed. Plus, it also supports full-text search.
Like ducky says, if you're going to query the column using a SQL LIKE, you're query is going to be very slow, no matter what index you put on the column.
There's 2 options:
Switch to MyIsam database instead of InnoDB and use full-text search on this column. This is done by placing a 'fulltext' index on the column. More Information.
Use a tool like Lucene for full-text searching