I have what seems like a fairly simple table structure, however MySQL is defaulting to a less than optimal index_merge on a simple query.
Here's the table structure:
CREATE TABLE IF NOT EXISTS `event_log` (
`event_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(5) DEFAULT NULL,
`location_id` int(10) DEFAULT NULL,
`object_id` int(5) DEFAULT NULL,
`action_id` int(5) DEFAULT NULL,
`date_event` datetime DEFAULT NULL,
PRIMARY KEY (`event_id`),
KEY `user_id` (`user_id`),
KEY `date_event` (`date_event`),
KEY `action_id` (`action_id`),
KEY `object_id` (`object_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
EXPLAIN on a basic SELECT query
EXPLAIN SELECT date_event
FROM event_log
WHERE user_id =123
AND object_id =456
AND location_id =789
Returns this:
select_type table type possible_keys key key_len ref rows Extra
SIMPLE event_log index_merge user_id,object_id object_id,user_id 5,5 NULL 27 Using intersect(object_id,user_id); Using where
Here's the Extra bit, for easier reading:
Using intersect(object_id,user_id); Using where
Why is MySQL not using standard indexes on this query? Why is it intersecting user_id and object_id?
The most effective index for the query is a composite index that includes all three fields, for example: (object_id, user_id, location_id). Since there is no such index, MySQL does its best to get most of the information from existing indexes.
Related
I have read several other posts and pages and i can't figure a way to speed up and use an index instead of a full table scan
Query in question
select f.*,ci.item_name
from forecastpro.current_item_forecast f
left join cat_items as ci on f.item_number=ci.item_number
Tables Structures
CREATE TABLE `current_item_forecast` (
`Axe3` varchar(128) DEFAULT NULL,
`Axe5` varchar(128) DEFAULT NULL,
`Axe7` varchar(128) DEFAULT NULL,
`Axe11` varchar(128) DEFAULT NULL,
`Item_Number` varchar(40) NOT NULL,
`Item_Name` varchar(128) DEFAULT NULL,
`fc_year` int(4) NOT NULL,
`fc_period` int(2) NOT NULL,
`Qty_Forecast` double NOT NULL DEFAULT '0',
`USD_Forecast` double NOT NULL DEFAULT '0',
`Stock_Start` float NOT NULL DEFAULT '0',
`Stock_Transit` float NOT NULL DEFAULT '0',
PRIMARY KEY (`Item_Number`,`fc_year` DESC,`fc_period`),
KEY `Item` (`Item_Number`),
KEY `Axe3` (`Axe3`),
KEY `Axe5` (`Axe5`),
KEY `Axe7` (`Axe7`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
CREATE TABLE `cat_items` (
`Item_Number` char(32) NOT NULL,
`Item_Name` char(128) DEFAULT NULL,
`Axe3` char(10) DEFAULT NULL,
`Item_Group` varchar(32) DEFAULT NULL,
`Axe5` char(10) DEFAULT NULL,
`Axe7` char(10) DEFAULT NULL,
`Axe11` char(10) DEFAULT NULL,
`Date_Created` datetime DEFAULT NULL,
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Explain
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE f ALL 121882 100.00
1 SIMPLE ci ALL 36838 100.00 Using where; Using join buffer (Block Nested Loop)
Other Attempts
**This two produce the same result:**
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci on f.item_number=ci.item_number
set f.item_name=ci.item_name
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci FORCE KEY FOR JOIN (PRIMARY) on f.item_number=ci.item_number
set f.item_name=ci.item_name
id select_type table type possible_keys key key_len ref rows Extra
1 UPDATE f ALL 76953 100.00
1 SIMPLE ci ALL 36838 100.00 Using where
Any attempt to run simply runs forever.
What would you recomend?
For your first select query, you may try adding the following index to the cat_items table:
CREATE INDEX idx ON cat_items (item_number, item_name);
This ideally should speed up the left join process, as each record in the current_item_forecast table gets matched to the cat_items table using the item_number as a lookup. Note that we also include the item_name in the index so as to cover the SELECT clause.
(Thank you for supplying the CREATE TABLEs; the root cause in your problem would have been very hard to figure out without them.)
Do not mix CHARACTER SETs, at least not when JOINing by varchars. It leads to a costly table scan.
current_item_forecast:
`Item_Number` varchar(40) NOT NULL ... DEFAULT CHARSET=utf8mb4
cat_items:
`Item_Number` char(32) ... DEFAULT CHARSET=utf8
Be consistent in lengths (40 vs 32).
Since PRIMARY KEY is a unique key, don't redundantly include KEY for the same column:
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
Use VARCHAR unless the column is truly fixed width.
I have a table with about 700.000 rows:
CREATE TABLE IF NOT EXISTS `ext_log_entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`action` varchar(8) NOT NULL,
`logged_at` datetime NOT NULL,
`object_id` varchar(32) DEFAULT NULL,
`object_class` varchar(255) NOT NULL,
`version` int(11) NOT NULL,
`data` longtext COMMENT '(DC2Type:array)',
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `log_date_lookup_idx` (`logged_at`),
KEY `log_user_lookup_idx` (`username`),
KEY `log_class_lookup_idx` (`object_class`),
KEY `log_version_lookup_idx` (`object_id`,`object_class`,`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1219777 ;
I try to run the following query:
SELECT n0_.id AS id0, n0_.action AS action1, n0_.logged_at AS logged_at2, n0_.object_id AS object_id3, n0_.object_class AS object_class4, n0_.version AS version5, n0_.data AS data6, n0_.username AS username7
FROM ext_log_entries n0_
WHERE n0_.object_id =275634
AND n0_.object_class = 'My\\MyBundle\\Entity\\Field'
AND n0_.version <=1
ORDER BY n0_.version ASC
Here is the MySQL plan:
id 1
select_type SIMPLE
table n0_
type ref
possible_keys log_class_lookup_idx,log_version_lookup_idx
key log_class_lookup_idx
key_len 767
ref const
rows 641159
Extra Using where; Using filesort
My query need about 37 seconds to be executed for only 1 row in the result...
I tried to run the same query by deleting my indexes and it goes a little bit faster : about 31 seconds...
I don't understand why my query is taking so much time and why my indexes don't help the performance? Do you know how I can do to have good performance on this query?
Thanks in advance for your help !
EDIT
Here are the cardinalties of the indexes
log_date_lookup_idx BTREE logged_at 1221578 A
log_user_lookup_idx BTREE username 40 A YES
log_class_lookup_idx BTREE object_class 1010 A
log_version_lookup_idx BTREE object_id 1221578 A YES
object_class 1221578 A
version 1221578 A
I found a solution, not THE solution, but at least it works for me.
I think it could help anyway all people who are using gedmo loggable and who are lucky (like me) to have objects with only integers IDs.
I changes my column object_id to integer instead of varchar(255). My query now take 0.008 second ! It works for me because i'm sure i'll always have only integers, for people who have varchar, I'm sorry i tried many things but nothing worked....
CREATE TABLE IF NOT EXISTS `ext_log_entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`action` varchar(8) NOT NULL,
`logged_at` datetime NOT NULL,
`object_id` int(11) DEFAULT NULL,
`object_class` varchar(255) NOT NULL,
`version` int(11) NOT NULL,
`data` longtext COMMENT '(DC2Type:array)',
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `log_date_lookup_idx` (`logged_at`),
KEY `log_user_lookup_idx` (`username`),
KEY `log_class_lookup_idx` (`object_class`),
KEY `log_version_lookup_idx` (`object_id`,`object_class`,`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1219777 ;
The following query takes 10 seconds to finish when having order by. Without order by it finish in 0.0005 seconds. I am already having an index on field "sku", "vid" AND "timestamp". I have more 200,000 record in this table. Please help, what is wrong with the query when using order by.
SELECT i.pn,i.sku,i.title, fl.f_inserted,fl.f_special, fl.f_notinserted
FROM inventory i
LEFT JOIN inventory_flags fl ON fl.sku = i.sku AND fl.vid = i.vid
WHERE i.qty >=2 ORDER BY i.timestamp LIMIT 0,100;
-- --------------------------------------------------------
--
-- Table structure for table `inventory`
--
CREATE TABLE IF NOT EXISTS `inventory` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`pn` varchar(60) DEFAULT NULL,
`sku` varchar(60) DEFAULT NULL,
`title` varchar(60) DEFAULT NULL,
`qty` int(11) DEFAULT NULL,
`vid` int(11) DEFAULT NULL,
`timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `vid` (`vid`),
KEY `sku` (`sku`),
KEY `timestamp` (`timestamp`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
-- --------------------------------------------------------
--
-- Table structure for table `inventory_flags`
--
CREATE TABLE IF NOT EXISTS `inventory_flags` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`f_inserted` tinyint(1) DEFAULT NULL,
`f_notinserted` tinyint(1) DEFAULT NULL,
`f_special` tinyint(1) DEFAULT NULL,
`timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`sku` varchar(60) DEFAULT NULL,
`vid` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `vid` (`vid`),
KEY `sku` (`sku`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
EXPLANE RESULT:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE fl system vid,sku NULL NULL NULL 0 const row not found
1 SIMPLE i index NULL timestamp 5 NULL 10 Using where
Instead of adding seprate indexes on columns you need to put multicolumn index on tables as you are using more than one columns from same table in joining condition.
after including columns from WHERE clause also include columns used in ORDER BY clause in composite index.
try adding flowing indexes and test them using EXPLAIN:
ALTER TABLE ADD INDEX ix_if inventory_flags(sku, vid);
ALTER TABLE ADD INDEX ix_i inventory(sku, qty, timestamp);
also try to avoid DISTINCT clause in your query, it is equivalent to GROUP BY clause, if you still need it then consider adding covering index.
If sku is unique to each inventory item then define it as UNIQUE - it'll speed things up. (Or the combination of sku and vid - define a composite index in that case.)
Why are you doing SELECT DISTINCT? The vast majority of the time using DISTINCT is a sign that your query or your table structure is wrong.
Since it's DISTINCT, and sku is not UNIQUE it can't use the index on timestamp to speed things up, so it has to sort a table with 200,000 records - it can't even use an index on qty to speed that part up.
PS. Omesh has some good advice as well.
you can use force index(index_key). try it, and you will see in explain query that mysql now will use the key index when 'order by'
table:
CREATE TABLE `deal` (
`id` int(11) NOT NULL default '0',
`site` int(11) NOT NULL default '0',
`time` bigint(13) NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `site` (`site`),
KEY `time` (`time`,`site`)
) TYPE=MyISAM
sql query:
select * from `deal` where time>0 && site=8
I create index:time for this query,
but why this query always using index: site?
explain select * from `deal` where time>0 && site=8
output:
table type possible_keys key key_len ref rows Extra
deal ref site,time site 4 const 1 Using where
You need to create composite index site + time (yes, order matters).
So delete both indexes site and time now and create 2:
KEY site (site, time)
KEY time (time)
I can't seem to get my query to stop using filesort.
This is my query:
SELECT s.`pilot`, p.`name`, s.`sector`, s.`hull`
FROM `pilots` p
LEFT JOIN `ships` s ON ( (s.`game` = p.`game`)
AND (s.`pilot` = p.`id`) )
WHERE p.`game` = 1
AND p.`id` <> 2
AND s.`sector` = 43
AND s.`hull` > 0
ORDER BY p.`last_move` DESC
Table structures:
CREATE TABLE IF NOT EXISTS `pilots` (
`id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`game` tinyint(3) unsigned NOT NULL DEFAULT '0',
`last_move` int(10) NOT NULL DEFAULT '0',
UNIQUE KEY `id` (`id`),
KEY `last_move` (`last_move`),
KEY `game_id_lastmove` (`game`,`id`,`last_move`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8 ;
CREATE TABLE IF NOT EXISTS `ships` (
`id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`game` tinyint(3) unsigned NOT NULL DEFAULT '0',
`pilot` mediumint(5) unsigned NOT NULL DEFAULT '0',
`sector` smallint(5) unsigned NOT NULL DEFAULT '0',
`hull` smallint(4) unsigned NOT NULL DEFAULT '50',
UNIQUE KEY `id` (`id`),
KEY `game` (`game`),
KEY `pilot` (`pilot`),
KEY `sector` (`sector`),
KEY `hull` (`hull`),
KEY `game_2` (`game`,`pilot`,`sector`,`hull`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8 ;
The explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE p ref id,game_id_lastmove game_id_lastmove 1 const 7 Using where; Using filesort
1 SIMPLE s ref game,pilot,sector... game_2 6 const,fightclub_alpha.p.id,const 1 Using where; Using index
edit: I cut some of the unnecessary pieces out of my queries/table structure.
Anybody have any ideas?
the best thing that you can do is to make indexes:
index that covers table ships with fields: game + pilot + sector + hull (in this specific order)
pilots: game + id
this particular query will always use filesort, because it has not range condition p.id <> 2
http://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html
In some cases, MySQL cannot use
indexes to resolve the ORDER BY,
although it still uses indexes to find
the rows that match the WHERE clause.
These cases include the following ...
The key used to fetch the rows is not
the same as the one used in the ORDER
BY