I have strange problem on query optimization. SQL is generated by ORM-like library and something bad has been detected only after reading megabytes of SQL logs.
SELECT
`ct_pricelistentry`.`uid` as `uid`, `ct_pricelistentry`.`skuGroup` as `skuGroup`
FROM
`ct_pricelistentry` INNER JOIN `lct_set`
ON `lct_set`.`parent_uid`='SET:ALLPRICELISTENTRIES' AND
`lct_set`.`ref_uid`=`ct_pricelistentry`.`uid`
WHERE
(`isGroup` IS FALSE) AND
(`isService` IS FALSE) AND
(`brand` = 'BRAND:5513f43697d637.00632331' OR `brand` IS NULL)
ORDER BY `skuGroup` ASC
EXPLAIN says:
'1', 'SIMPLE', 'ct_pricelistentry', 'ALL', 'PRIMARY', NULL, NULL, NULL, '34591', 'Using where; Using filesort'
'1', 'SIMPLE', 'lct_set', 'eq_ref', 'PRIMARY', 'PRIMARY', '292', 'const,dealers_v2.ct_pricelistentry.uid', '1', 'Using where; Using index'
Note: all needed indexes are presented including skuGroup. But the index skuGroupstill is not listed in EXPLAIN possible_keys. It also cannot be forced by FORCE INDEX (it just disables all indexes).
After some research I found hacky solution but not sure it works as indended:
Add FORCE INDEX (skuGroup),
Add to the WHERE clause dummy AND (skuGroup IS NULL OR skuGroup IS NOT NULL) part.
Following query
SELECT
`ct_pricelistentry`.`uid` as `uid`, `ct_pricelistentry`.`skuGroup` as `skuGroup`
FROM
`ct_pricelistentry` FORCE INDEX (`skuGroup`) INNER JOIN `lct_set`
ON `lct_set`.`parent_uid`='SET:ALLPRICELISTENTRIES' AND
`lct_set`.`ref_uid`=`ct_pricelistentry`.`uid`
WHERE
(`isGroup` IS FALSE) AND
(`isService` IS FALSE) AND
(`brand` = 'BRAND:5513f43697d637.00632331' OR `brand` IS NULL) AND
(`skuGroup` IS NULL OR `skuGroup` IS NOT NULL)
ORDER BY `skuGroup` ASC
gives EXPLAIN without filesort so it seems to use the index to fetch ordered rows:
'1', 'SIMPLE', 'ct_pricelistentry', 'range', 'skuGroup', 'skuGroup', '768', NULL, '16911', 'Using where'
'1', 'SIMPLE', 'lct_set', 'eq_ref', 'PRIMARY', 'PRIMARY', '292', 'const,dealers_v2.ct_pricelistentry.uid', '1', 'Using where; Using
index'
Whats happening at all?
Is it a MySQL bug? I've tested on MySQL 5.1 - 5.5 - the same results. Do you have more predictable/stable solutions?
---- CREATE TABLE ----
CREATE TABLE IF NOT EXISTS `lct_set` (
`parent_uid` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`ref_uid` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`parent_uid`,`ref_uid`),
UNIQUE KEY `BACK_PRIMARY` (`ref_uid`,`parent_uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `ct_pricelistentry` (
`uid` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`refcount` int(11) NOT NULL,
`isDisposed` tinyint(1) DEFAULT NULL,
`tag` text,
`isGroup` tinyint(1) DEFAULT NULL,
`parentEntry` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`externalUID` varchar(255) DEFAULT NULL,
`productCode` varchar(16) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`sku` varchar(255) DEFAULT NULL,
`skuGroup` varchar(255) DEFAULT NULL,
`measureUnit` varchar(16) DEFAULT NULL,
`image` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`itemClassExternalUID` varchar(255) DEFAULT NULL,
`itemClassName` varchar(255) DEFAULT NULL,
`itemClassDescription` text,
`itemClassComments` text,
`itemClassAttachments` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`brand` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`priceGroups` text,
`productAttributes` text,
`constituents` text,
`position` int(11) DEFAULT NULL,
`isService` tinyint(1) DEFAULT NULL,
`stackability` varchar(255) DEFAULT NULL,
PRIMARY KEY (`uid`),
UNIQUE KEY `test1` (`uid`,`skuGroup`),
KEY `name` (`name`),
KEY `sku` (`sku`),
KEY `itemClassExternalUID` (`itemClassExternalUID`),
KEY `parentEntry` (`parentEntry`),
KEY `position` (`position`),
KEY `externalUID` (`externalUID`),
KEY `productCode` (`productCode`),
KEY `skuGroup` (`skuGroup`),
KEY `brand` (`brand`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The fix Using the INDEX(skuGroup) avoids the filesort, but prevents any useful filtering. Optimizing the filtering is more important than avoiding the filesort.
Remove the FORCE and add this 'composite' index
INDEX(isGroup, isService, brand) -- (in any order)
It should help, but probably won't prevent "using filesort". The OR is the killer.
To prevent the use of filesort for ORDER BY, you need a single (usually composite) index that includes all of the WHERE clause, plus the ORDER BY column(s). In constructing such an index, the only things that can be handled in the WHERE are and'd together '=' clauses. Anything else (such as your OR) prevents the optimization.
Why OR hurts Think of it this way... Suppose there were a long printed list of names sorted by last name + first name. And the query asked for WHERE last = 'Karakulov' ORDER BY first. You would jump to the first Karakulov, and there would be all the first names in order. Now suppose you wanted WHERE (last = 'Karakulov' OR last = 'James') ORDER BY first. You could get all your relatives and all my relatives, but you still need to shuffle them together to do the ORDER BY first. MySQL has one technique for that: filesort (and a tmp table leading up to it.)
As a consolation, filesort's temp table is usually an in-memory MEMORY table, so it is reasonably fast.
A workaround is sometimes to turn OR into UNION. (It probably would not help for your query.)
Some schema critique, and other notes...
The UNIQUE key is useless because the PRIMARY KEY already declares uid to be "unique".
VARCHAR(48) utf8 is a rather clumsily big key. Is it some form of UUID? If so, I have nasty things to say about randomness and charset and size.
Some uids are (48), some are (255); was this deliberate?
Get rid of (skuGroupIS NULL ORskuGroupIS NOT NULL) -- The Optimizer is probably not smart enough to realize that this is always "TRUE" !
FORCE INDEX may work today, but could backfire tomorrow. Get rid of it.
What is the value of innodb_buffer_pool_size? It should be about 70% of available RAM if you have at least 4GB of ram. If you left it at some low default, then you are probably I/O-bound, hence slow.
Please provide SHOW CREATE TABLE lct_set -- something strange is going on in the JOIN.
Related
my app get stuck for hours on simple queries like :
SELECT COUNT(*) FROM `item`
Context :
This table is around 200Gb+ and 50M+ rows.
We have a RDS on AWS with 2CPU and 16GiB RAM (db.r6g.large).
This is the table structure SQL dump :
/*
Target Server Type : MySQL
Target Server Version : 80023
File Encoding : 65001
*/
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
DROP TABLE IF EXISTS `item`;
CREATE TABLE `item` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`status` tinyint DEFAULT '1',
`source_id` int unsigned DEFAULT NULL,
`type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`url` varchar(2048) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`title` varchar(500) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`sku` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`price` decimal(20,4) DEFAULT NULL,
`price_bc` decimal(20,4) DEFAULT NULL,
`price_original` decimal(20,4) DEFAULT NULL,
`currency` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`description` text CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
`image` varchar(1024) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`time_start` datetime DEFAULT NULL,
`time_end` datetime DEFAULT NULL,
`block_update` tinyint(1) DEFAULT '0',
`status_api` tinyint(1) DEFAULT '1',
`data` json DEFAULT NULL,
`created_at` int unsigned DEFAULT NULL,
`updated_at` int unsigned DEFAULT NULL,
`retailer_id` int DEFAULT NULL,
`hash` char(32) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`count_by_hash` int DEFAULT '1',
`item_last_update` int DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `sku_retailer_idx` (`sku`,`retailer_id`),
KEY `updated_at_idx` (`updated_at`),
KEY `time_end_idx` (`time_end`),
KEY `retailer_id_idx` (`retailer_id`),
KEY `hash_idx` (`hash`),
KEY `source_id_hash_idx` (`source_id`,`hash`) USING BTREE,
KEY `count_by_hash_idx` (`count_by_hash`) USING BTREE,
KEY `created_at_idx` (`created_at`) USING BTREE,
KEY `title_idx` (`title`),
KEY `currency_idx` (`currency`),
KEY `price_idx` (`price`),
KEY `retailer_id_title_idx` (`retailer_id`,`title`) USING BTREE,
KEY `source_id_idx` (`source_id`) USING BTREE,
KEY `source_id_count_by_hash_idx` (`source_id`,`count_by_hash`) USING BTREE,
KEY `status_idx` (`status`) USING BTREE,
CONSTRAINT `fk-source_id` FOREIGN KEY (`source_id`) REFERENCES `source` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1858202585 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
SET FOREIGN_KEY_CHECKS = 1;
does partitioning the table could help on a simple query like this ?
do I need to increase the RAM of the RDS ? If yes what configuration do I need ?
is NoSQL more compatible to this kind of structure ?
Do you have any advices/solutions/fixes so the app can run those queries (we would like to keep all the data and not erase it if possible..) ?
"SELECT COUNT(*) FROM item" needs to scan an index. The smallest index is about 200MB, so that seems like it should not take "minutes".
There are probably multiple queries that do full table scans. Such will bump out all the cached data from the ~11GB of cache (the buffer_pool) and do that about 20 times. That's a lot of I/O and a lot of elapsed time. Meanwhile, most other queries will run slowly because their cached data is being bumped out.
The resolution:
Locate these naughty queries. RDS probably gives you access to the "slowlog".
Grab the slowlog and run pt-query-digest or mysqldumpslow -s t to find the "worst" queries.
Then we can discuss them.
There are some redundant indexes; removing them won't solve the problem. A rule: If you have INDEX(a), INDEX(a,b), you don't need the former.
If hash is some kind of scrambled value, it is likely that a single-row lookup (or update) will require a disk hit (and bump something else out of the cache).
decimal(20,4) takes 10 bytes and allows values up to 9,999,999,999,999,999.9999; that seems excessive. (Shrinking it won't save much space; something to keep in mind for the future.)
I see that AUTO_INCREMENT has reached 1.8 billion. If there are only 50M rows, does the processing do a lot of DELETEs? Or maybe REPLACE``? IODKU is better than REPLACE`.
Thanks for all the advices here, but the problem was that we were using the MySQL json type for a very heavy column. Removing this column or even changing it to varchar made the COUNT(id) around 1000x faster (also adding WHERE id > 1 helped..)
Note : it was impossible to just delete the column as it was, we had to change it to varchar before.
I have following table - 'element'
CREATE TABLE `element` (
`eid` bigint(22) NOT NULL AUTO_INCREMENT,
`tag_name` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`text` text COLLATE utf8_bin,
`depth` tinyint(2) DEFAULT NULL,
`classes` tinytext COLLATE utf8_bin,
`webarchiver_uniqueid` int(11) DEFAULT NULL,
`created` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
`rowstatus` char(1) COLLATE utf8_bin DEFAULT 'A',
PRIMARY KEY (`eid`)
) ENGINE=InnoDB AUTO_INCREMENT=12090 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Column details and current index details are given above. Almost 90% of queries on this table are like:
select * from element
where tag_name = 'XXX'
and text = 'YYYY'
and depth = 20
and classes = 'ZZZZZ'
and rowstatus = 'A'
What would be the most optimal way to create index on this table? The table has around 60k rows.
Change classes from TINYTEXT to VARCHAR(255) (or some more reasonable size), then have
INDEX(tag_name, depth, classes)
with the columns in any order. I left out rowstatus because it smells like a column that is likely to change. (Anyway, a flag does not add much to an index.)
You can't include TEXT or BLOB columns in an index. And it is not worth it to do a 'prefix' index.
Since a PRIMARY KEY is a UNIQUE key, DROP INDEX eid_UNIQUE.
Is there some reason for picking "binary" / "utf8_bin" for all the character fields?
The problem is that after I insert 200.000-300.000 rows of data into those columns the search moves very slow and my first thing that came in mind is the indexes that I may have not added correctly. I have tried adding as many as possible BTREE indexes phpmyadmin did not let me to add for all. What would be the correct indexes for my table? I have the following table with the following indexes:
CREATE TABLE IF NOT EXISTS `carads` (
`ADID` int(7) NOT NULL AUTO_INCREMENT,
`LINK` varchar(255) CHARACTER SET latin1 NOT NULL,
`TITLE` varchar(255) NOT NULL,
`MAKE` varchar(50) CHARACTER SET latin1 NOT NULL,
`MODEL` varchar(100) CHARACTER SET latin1 NOT NULL,
`FUEL` varchar(50) CHARACTER SET latin1 NOT NULL,
`LOC` varchar(100) NOT NULL,
`TRANS` varchar(50) NOT NULL,
`YEAR` varchar(4) CHARACTER SET latin1 NOT NULL,
`BODY` varchar(255) CHARACTER SET latin1 NOT NULL,
`DESCRIPT` text CHARACTER SET latin1 NOT NULL,
`PHONENR` varchar(20) NOT NULL,
`MILEAGE` int(11) NOT NULL,
`PRICE` int(20) NOT NULL,
`DISTANCE` int(250) NOT NULL,
`POSTCODE` varchar(250) NOT NULL,
`IMAGE1` varchar(255) NOT NULL,
`IMAGE2` varchar(255) NOT NULL,
`IMAGE3` varchar(255) NOT NULL,
`IMAGE4` varchar(255) NOT NULL,
`IMAGE5` varchar(255) NOT NULL,
`CPHONE` varchar(250) NOT NULL,
`CEMAIL` varchar(500) NOT NULL,
`COLOUR` varchar(250) NOT NULL,
`EQUIPMENT` text NOT NULL,
`STATUS` tinyint(1) NOT NULL DEFAULT '1',
`DATE` date NOT NULL,
`DEL` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`ADID`),
KEY `ix_MakeModelPrice` (`STATUS`,`MAKE`(25),`MODEL`(25),`PRICE`),
KEY `ix_Price` (`PRICE`,`STATUS`,`DEL`,`TITLE`(30),`ADID`),
KEY `ix_Date` (`DATE`,`STATUS`,`DEL`,`TITLE`(30),`ADID`),
KEY `LINK` (`LINK`),
FULLTEXT KEY `MODEL` (`MODEL`),
FULLTEXT KEY `SearchIndex` (`TITLE`,`LOC`,`TRANS`,`CPHONE`,`CEMAIL`,`COLOUR`,`EQUIPMENT`),
FULLTEXT KEY `MAKE` (`MAKE`)
)
ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=2478687;
This is very complicated and we cannot give you the correct answer, you have to understand and find the best answer by yourself.
You have to keep following in mind:
The query optimizer will choose only one index.
Indexes which start with something like "status" and or "del" (boolean values or values where 95% of the rows have the selected values) don't add any value, besides these dummy columns are followed with often queried, highly selective values.
You should first find the attributes which are
filled in most of the queries (I could imagine that "make", "price" and "year" are good candidates)
are most selective (meaning that the resulting rows are < 10%)
You have to find out which distribution of values for each of the columns exist in your table. Examples:
Make:
BMW: 5%
Alfa Romeo: 1%
VW: 7%
...
Price-Range:
0..999: 3%
1000..1999: 4%
2000..3000: 5%
...
If 80% of all searches contain "make", "price" and "year", then create an index with all 3 columns. Put the columns which are most selective and/or are mentioned in most searches to the front, followed by the other columns.
With some luck you can improve response time of many searches dramatically. You can then dig deeper into statistics and add some other indexes. Maybe 80% of all searches have a selection for "make", but in the rest there are still many searches without "make", but with focus on "price" and "fuel", then create an index for those searches.
You could as well improve performance when you use "codes" (e.g. Alfa Romeo=1, BMW=2, VW=3, ...) or cluster ranges of values (e.g. price_range: 0..999, 1000..2000, ...). This could help MySQL to build a bit more efficient indexes (smaller leads to less memory footprint and less I/0).
And to understand indexes better, try to submit a query like this (I want that index ix_MakeModelPrice is used):
-- ix_MakeModelPrice: STATUS`,`MAKE`(25),`MODEL`(25),`PRICE`
SELECT * FROM carads
where STATUS=1 AND MAKE='Alfa Romeo'
AND MODEL='159' and PRICE BETWEEN 100 and 1000
order by ADID Desc
LIMIT 0
This query should be fast (hopefully with some matching rows). Do you see why it is fast? "STATUS" is not selective, but the rest should reduce the number of rows found with an index-scan to probably way below 1%. The number of physical reads (rows) is reduced to a minimum => faster response.
SELECT fcat.id,fcat.title,fcat.description,
count(DISTINCT ftopic.id) as number_topics,
count(DISTINCT fpost.id) as number_posts FROM fcat
LEFT JOIN ftopic ON fcat.id=ftopic.cat_id
LEFT JOIN fpost ON ftopic.id=fpost.topic_id
GROUP BY fcat.id
ORDER BY fcat.ord
LIMIT 100;
index on ftopic_cat_id, fpost.topic_id, fcat.ord
EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE fcat ALL PRIMARY NULL NULL NULL 11 Using temporary; Using filesort
1 SIMPLE ftopic ref PRIMARY,cat_id_2 cat_id_2 4 bloki.fcat.id 72
1 SIMPLE fpost ref topic_id_2 topic_id_2 4 bloki.ftopic.id 245
fcat - 11 rows,
ftopic - 1106 rows,
fpost - 363000 rows
Query takes 4,2 sec
TABLES:
CREATE TABLE IF NOT EXISTS `fcat` (
`id` int(11) NOT NULL auto_increment,
`title` varchar(250) collate utf8_unicode_ci NOT NULL,
`description` varchar(250) collate utf8_unicode_ci NOT NULL,
`created` datetime NOT NULL,
`visible` tinyint(4) NOT NULL default '1',
`ord` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `ord` (`ord`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=12 ;
CREATE TABLE IF NOT EXISTS `ftopic` (
`id` int(11) NOT NULL auto_increment,
`cat_id` int(11) NOT NULL,
`title` varchar(100) collate utf8_unicode_ci NOT NULL,
`created` datetime NOT NULL,
`updated` timestamp NOT NULL default CURRENT_TIMESTAMP,
`lastname` varchar(200) collate utf8_unicode_ci NOT NULL,
`visible` tinyint(4) NOT NULL default '1',
`closed` tinyint(4) NOT NULL default '0',
`views` int(11) NOT NULL default '1',
PRIMARY KEY (`id`),
KEY `cat_id_2` (`cat_id`,`updated`,`visible`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1116 ;
CREATE TABLE IF NOT EXISTS `fpost` (
`id` int(11) NOT NULL auto_increment,
`topic_id` int(11) NOT NULL,
`pet_id` int(11) NOT NULL,
`content` text collate utf8_unicode_ci NOT NULL,
`imageName` varchar(300) collate utf8_unicode_ci NOT NULL,
`created` datetime NOT NULL,
`reply_id` int(11) NOT NULL,
`visible` tinyint(4) NOT NULL default '1',
`md5` varchar(100) collate utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `md5` (`md5`),
KEY `topic_id_2` (`topic_id`,`created`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=390971 ;
Thanks,
hamlet
you need to create a key with both fcat.id, fcat.ord
Bold rewrite
This code is not functionally identical, but...
Because you want to know about distinct ftopic.id and fpost.id I'm going to be bold and suggest two INNER JOIN's instead of LEFT JOIN's.
Then because the two id's are autoincrementing they will no longer repeat, so you can drop the distinct.
SELECT
fcat.id
, fcat.title
, fcat.description
, count(ftopic.id) as number_topics
, count(fpost.id) as number_posts
FROM fcat
INNER JOIN ftopic ON fcat.id = ftopic.cat_id
INNER JOIN fpost ON ftopic.id = fpost.topic_id
GROUP BY fcat.id
ORDER BY fcat.ord
LIMIT 100;
It depends on your data if this is what you are looking for, but I'm guessing it will be faster.
All your indexes seem to be in order though.
MySQL does not use indexes for small sample sizes!
Note that the explain list that MySQL only has 11 rows to consider for fcat. This is not enough for MySQL to really start worrying about indexes, so it doesn't.
Because going to the index for small row-counts slows things down.
MySQL is trying to speed things up so it chooses not to use the index, this confuses a lot of people because we are trained so hard on the index. Small sample sizes don't give good explains!
Increase the size of the test data so MySQL has more rows to consider and you should start seeing the index being used.
Common misconceptions about force index
Force index does not force MySQL to use an index as such.
It hints at MySQL to use a different index from the one it might naturally use and it pushes MySQL into using an index by setting a very high cost on a table scan.
(In your case MySQL is not using a table scan, so force index has no effect)
MySQL (same most other DBMS's on the planet) has a very strong urge to use indexes, so if it doesn't (use any) that's because using no index at all is faster.
How does MySQL know which index to use
One of the parameters the query optimizer uses is the stored cardinality of the indexes.
Over time these values change... But studying the table takes time, so MySQL doesn't do that unless you tell it to.
Another parameter that affects index selection is the predicted disk-seek-times that MySQL expects to encounter when performing the query.
Tips to improve index usage
ANALYZE TABLE will instruct MySQL to re-evaluate the indexes and update its key distribution (cardinality). (consider running it daily/weekly in a cron job)
SHOW INDEX FROM table will display the key distribution.
MyISAM tables and indexes fragment over time. Use OPTIMIZE TABLE to unfragment the tables and recreate the indexes.
FORCE/USE/IGNORE INDEX limits the options MySQL's query optimizer has to perform your query. Only consider it on complex queries.
Time the effect of your meddling with indexes on a regular basis. A forced index that speeds up your query today might slow it down tomorrow because the underlying data has changed.
How do I get MySQL to use a key/index with the following table structure and query?
-- the table
CREATE TABLE `country` (
`id` int(11) NOT NULL auto_increment,
`expiry_date` datetime NOT NULL,
`name` varchar(50) collate utf8_unicode_ci NOT NULL,
`symbol` varchar(5) collate utf8_unicode_ci NOT NULL,
`exchange_rate` decimal(11,5) NOT NULL default '1.00000',
`code` char(3) collate utf8_unicode_ci NOT NULL,
`currency_code` varchar(3) collate utf8_unicode_ci NOT NULL,
`display_order` smallint(6) unsigned NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `code` (`code`),
KEY `currency_code` (`currency_code`),
KEY `display_order` (`expiry_date`,`name`,`display_order`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
-- the query
SELECT `country`.*
FROM `country`
WHERE `country`.`expiry_date` = 0
ORDER BY `country`.`display_order` ASC, `country`.`name` ASC;
I'm trying to get it to use a key because the query with 180 in the result takes 0.0013s and is by far the slowest query on the page (3x longer than the next slowest). From my understanding, the query should use the display_order index/key.
Change it to:
CREATE TABLE `country` (
`id` int(11) NOT NULL auto_increment,
`expiry_date` datetime NOT NULL,
`name` varchar(50) collate utf8_unicode_ci NOT NULL,
`symbol` varchar(5) collate utf8_unicode_ci NOT NULL,
`exchange_rate` decimal(11,5) NOT NULL default '1.00000',
`code` char(3) collate utf8_unicode_ci NOT NULL,
`currency_code` varchar(3) collate utf8_unicode_ci NOT NULL,
`display_order` smallint(6) unsigned NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `code` (`code`),
KEY `currency_code` (`currency_code`),
KEY `expiry` (`expiry_date`,`name`,`display_order`) <<- renamed key for clarity
/* always name compound keys for their left-most parts*/
KEY `name` (`name`) <<-- new key here
KEY `display` (`display_order`) <<--new key here
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
-- the query
SELECT `country`.*
FROM `country`
WHERE `country`.`expiry_date` = 0
ORDER BY `country`.`display_order` ASC, `country`.`name` ASC;
Compound indexes are tricky
MySQL did not use the index on name in the compound index, because name was in the middle and MySQL only uses parts of an index if that part is the left-most part of a compound index.
The same goes for the index on field display order. The compound index that has display_order in it uses that field as it's right-most part, and therefore will not sort.
Solution
Make a separate index for field name,
and a separate index for field display_order.
Sometimes MySQL does not use an index, even if one is available. One circumstance under which this occurs is when the optimizer estimates that using the index would require MySQL to access a very large percentage of the rows in the table. (In this case, a table scan is likely to be much faster because it requires fewer seeks.) However, if such a query uses LIMIT to retrieve only some of the rows, MySQL uses an index anyway, because it can much more quickly find the few rows to return in the result.
Also if a large percentage of rows have the same value for a field (> 40% (IIRC)) then MySQL will not use the index.
See: http://dev.mysql.com/doc/refman/5.1/en/mysql-indexes.html
See: http://dev.mysql.com/doc/refman/5.1/en/index-hints.html
On how to force indexes as per FractalizeR suggestion.
Make sure to time your select after forcing the index
On such a simple query MySQL seems unlikely to be wrong, and your select time of 0.0013 seconds suggests that there are few rows in the table.
Indexes don't work as you'd expect when there are few rows in a table, because of the percentage rule stated above.
Note that in this case forcing the index would not have worked, because you cannot force MySQL to use the rightmost part of a compound index. It just cannot do that.
If you think MySQL chooses indexes unwisely and you are sure of that, use FORCE INDEX index hint: http://dev.mysql.com/doc/refman/5.1/en/index-hints.html
Your query has an ORDER BY on columns {display_order}+{name}, while
your index named "display_order" is in fact defined on columns {expiry_date}+{name}+{display_order}.
The order of columns in the index does matter. You can benefit an index if you need sorting of filtering on columns that are the beginning of the index.
This become obvious if you keep in mind that index are pre-sorted information.
If you want to benefit an index on {display_order}+{name} then you need an index that begins with {display_order}+{name}. For example {display_order}+{name} or {display_order}+{name}+{expiry_date}.
So in order to optimize your query, you have to change your index in the table, or your SORT clause in the query.
last thing you can do is, use "FORCE INDEX" as mentionten by fractalizeR