JOIN very slow when using RIGHT JOIN on this query - mysql

I'm having a problem with this query that takes several seconds to complete. I already tried many optimizations but I'm shooting blanks at this point.
The tables are the following (and are not absolutely normalized fully especially the tracks table)
CREATE TABLE `tracks` (
`id` int(14) unsigned NOT NULL AUTO_INCREMENT,
`artist` varchar(200) NOT NULL,
`track` varchar(200) NOT NULL,
`album` varchar(200) NOT NULL,
`path` text NOT NULL,
`tags` text NOT NULL,
`priority` int(10) NOT NULL DEFAULT '0',
`lastplayed` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`lastrequested` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`usable` int(1) NOT NULL DEFAULT '0',
`accepter` varchar(200) NOT NULL DEFAULT '',
`lasteditor` varchar(200) NOT NULL DEFAULT '',
`hash` varchar(40) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `hash` (`hash`),
FULLTEXT KEY `searchindex` (`tags`,`artist`,`track`,`album`),
FULLTEXT KEY `artist` (`artist`,`track`,`album`,`tags`)
) ENGINE=MyISAM AUTO_INCREMENT=3336 DEFAULT CHARSET=utf8
CREATE TABLE `esong` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`hash` varchar(40) COLLATE utf8_bin NOT NULL,
`len` int(10) unsigned NOT NULL,
`meta` text COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `hash` (`hash`)
) ENGINE=InnoDB AUTO_INCREMENT=16032 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
CREATE TABLE `efave` (
`id` int(10) unsigned NOT NULL DEFAULT '0',
`inick` int(10) unsigned NOT NULL,
`isong` int(10) unsigned NOT NULL,
UNIQUE KEY `inick` (`inick`,`isong`),
KEY `isong` (`isong`),
CONSTRAINT `inick` FOREIGN KEY (`inick`) REFERENCES `enick` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `isong` FOREIGN KEY (`isong`) REFERENCES `esong` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8
CREATE TABLE `enick` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT
`nick` varchar(30) COLLATE utf8_bin NOT NULL,
`dta` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`dtb` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
KEY `nick` (`nick`)
) ENGINE=InnoDB AUTO_INCREMENT=488 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
and the query I'm trying to execute with a normal speed is the following
SELECT esong.meta, tracks.id FROM tracks RIGHT JOIN esong ON tracks.hash = esong.hash JOIN efave ON efave.isong = esong.id JOIN enick ON efave.inick = enick.id WHERE enick.nick = lower('nickname');
Where if you remove the RIGHT JOIN and change it to JOIN it is fast
The EXPLAIN gives me this result, it seems there is a small problem in the efave selection but I have no idea how to get that out
+----+-------------+--------+--------+---------------+---------+---------+-----------------------+------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+--------+---------------+---------+---------+-----------------------+------+----------+--------------------------+
| 1 | SIMPLE | enick | ref | PRIMARY,nick | nick | 92 | const | 1 | 100.00 | Using where; Using index |
| 1 | SIMPLE | efave | ref | inick,isong | inick | 4 | radiosite.enick.id | 12 | 100.00 | Using index |
| 1 | SIMPLE | esong | eq_ref | PRIMARY | PRIMARY | 4 | radiosite.efave.isong | 1 | 100.00 | |
| 1 | SIMPLE | tracks | ALL | hash | NULL | NULL | NULL | 3210 | 100.00 | |
+----+-------------+--------+--------+---------------+---------+---------+-----------------------+------+----------+--------------------------+

Your explain looks clean, the only thing that stands out to me is the fact that the esong table is using a collate of utf8_bin, and the tracks table doesn't have a collation specified, which means it is probably using another collation type. Try aligning your collations and see how the join performs.

Have you checked your Execution Plan? If not, run your query to include it. Your Right Join may be doing an Index Scan instead of an Index Seek. Or you may be lacking indexes. Either way, you need to look at your Execution Plan so you can optimize your query better. No one will really be able to tell you how to make it faster using a Right Join (or a Join for that matter) until you know what the real problem is. Here are some links..
For MySQL: http://dev.mysql.com/doc/refman/5.5/en/execution-plan-information.html
For SqlServer: http://www.sql-server-performance.com/2006/query-execution-plan-analysis/

Related

Row Constructor Expression selects not using index

we have this table
CREATE TABLE `resource_grant` (
`resource_grant_id` int(11) NOT NULL AUTO_INCREMENT,
`member_type_id` int(11) NOT NULL,
`member_ref` varchar(36) NOT NULL,
`resource_type_id` int(11) NOT NULL,
`resource_ref` varchar(36) NOT NULL,
`role_id` int(11) NOT NULL,
`modified_by` varchar(255) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
`parent_grant_id` int(11) DEFAULT NULL,
PRIMARY KEY (`resource_grant_id`),
UNIQUE KEY `member_ref` (`member_ref`,`member_type_id`,`resource_type_id`,`resource_ref`),
KEY `member_type_id` (`member_type_id`),
KEY `resource_type_id` (`resource_type_id`),
KEY `role_id` (`role_id`),
KEY `resource_ref` (`resource_ref`,`resource_type_id`),
KEY `idx_rg_parent_grant_id` (`parent_grant_id`),
KEY `resource_ref_2` (`resource_ref`,`member_ref`,`resource_type_id`,`member_type_id`,`role_id`),
CONSTRAINT `resource_grant_ibfk_1` FOREIGN KEY (`member_type_id`) REFERENCES `member_type` (`member_type_id`),
CONSTRAINT `resource_grant_ibfk_2` FOREIGN KEY (`resource_type_id`) REFERENCES `resource_type` (`resource_type_id`),
CONSTRAINT `resource_grant_ibfk_3` FOREIGN KEY (`role_id`) REFERENCES `role` (`role_id`)
) ENGINE=InnoDB;
and these related tables
CREATE TABLE `member_type` (
`member_type_id` int(11) NOT NULL AUTO_INCREMENT,
`member_type` varchar(36) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`member_type_id`),
UNIQUE KEY `member_type` (`member_type`),
KEY `member_type_2` (`member_type`)
) ENGINE=InnoDB;
CREATE TABLE `resource_type` (
`resource_type_id` int(11) NOT NULL AUTO_INCREMENT,
`resource_type` varchar(36) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`resource_type_id`),
UNIQUE KEY `resource_type` (`resource_type`),
KEY `resource_type_2` (`resource_type`)
) ENGINE=InnoDB;
CREATE TABLE `role` (
`role_id` int(11) NOT NULL AUTO_INCREMENT,
`role_ref` varchar(50) NOT NULL,
`name` varchar(256) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`role_id`),
UNIQUE KEY `role_ref` (`role_ref`),
KEY `role_ref_2` (`role_ref`)
) ENGINE=InnoDB;
and we need to run selects like these ("Row Constructor Expression" syntax) (basically "bulk selects")
SELECT rg.resource_grant_id
FROM resource_grant rg
JOIN resource_type rt ON rg.resource_type_id = rt.resource_type_id
JOIN member_type mt ON rg.member_type_id = mt.member_type_id
JOIN role r ON r.role_id = rg.role_id
WHERE
(rg.resource_ref, rg.member_ref, rt.resource_type, mt.member_type, r.role_ref)
IN
(
('759','624962','property','epc-user','role.171'),
('11974','624962','property','epc-user','role.171')
);
the selects take ~60s to run, which is unacceptably long
note that there IS an index for (resource_ref,member_ref,resource_type_id,member_type_id,role_id)
we also don't want to run n individual select statements - we need these "bulk selects".
mysql 5.6 docs talk about this style of select not using indexes but you can make it using some tricks
https://dev.mysql.com/doc/refman/5.6/en/row-constructor-optimization.html
https://dev.mysql.com/doc/refman/5.6/en/range-optimization.html
not sure what's missing for us in order to make it use the indexes
EDIT here's the plan
mysql> explain SELECT rg.resource_grant_id FROM resource_grant rg JOIN resource_type rt ON rg.resource_type_id = rt.resource_type_id JOIN member_type mt ON rg.member_type_id = mt.member_type_id JOIN role r ON r.role_id = rg.role_id WHERE (rg.resource_ref, rg.member_ref, rt.resource_type, mt.member_type, r.role_ref) IN ( ('759','624962','property','epc-user','role.171'), ('11974','624962','property','epc-user','role.171') );
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
| 1 | SIMPLE | rt | index | PRIMARY | resource_type | 38 | NULL | 3 | Using index |
| 1 | SIMPLE | mt | index | PRIMARY | member_type | 38 | NULL | 6 | Using index; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | rg | ref | member_type_id,resource_type_id,role_id | member_type_id | 4 | samsDB.mt.member_type_id | 2370237 | Using where |
| 1 | SIMPLE | r | eq_ref | PRIMARY | PRIMARY | 4 | samsDB.rg.role_id | 1 | Using where |
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
4 rows in set (0.53 sec)
Start by changing the where clause to:
WHERE rg.member_ref = '624962' AND
rt.resource_type = 'property' AND
mt.member_type = 'epc-user' AND
r.role_ref = 'role.171' AND
rg.resource_ref IN ('759', '11974')
The existing indexes are not quite optimal for this. You need an index where the first two keys are (member_ref, resource_ref) -- well, except in the most recent versions of MySQL which implement skip-scan index optimizations.
You might be able to change resource_ref_2 to:
KEY `resource_ref_2` (`member_ref`, `resource_ref`, `resource_type_id`, `member_type_id`, `role_id`),
I'm not surprised at 60s on 5.6. "Row constructors" have existed for a long time. But they were not optimized before 5.7.
Either upgrade or rewrite the WHERE as Gordon suggests.

MYSQL request optimization select by simple column

I have a query like
SELECT `table1`.*
FROM `table1`
WHERE `table1`.`table2_id` IN (1,2,6,12,53,666)
and it works more than 20 seconds
Explain looks like:
+----+-------------+--------------------------+------------+-------+-------------------------------------------------------------------------------+----------------------------------+---------+------+-------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+-------+-------------------------------------------------------------------------------+----------------------------------+---------+------+-------+----------+-----------------------+
| 1 | SIMPLE | table1 | NULL | range | table2_id | table2_id | 4 | NULL | 74778 | 100.00 | Using index condition |
+----+-------------+--------------------------+------------+-------+-------------------------------------------------------------------------------+----------------------------------+---------+------+-------+----------+-----------------------+
Table looks like
CREATE TABLE `table1` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`table2_id` int(11) NOT NULL,
`table3_id` int(11) NOT NULL,
`field1` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`field2` int(11) NOT NULL DEFAULT '0',
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`field3` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `name_of_index_id` (`table3_id`),
KEY `other_name_of_index` (`field2`),
KEY `table2_id` (`table2_id`)
) ENGINE=InnoDB AUTO_INCREMENT=86623178 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
You can use index to fetch the result quickly. But, Indexing will take more space to generate and store in the database. So, if you are fine to utilize speed compare to space, Indexing will be used with following SQL statement.
ALTER TABLE `table1` ADD INDEX(`table2_id`);

Optimize MySQL Query - with a join within a sub select within a join

Okay, so there is a slow query in SugarCRM that is causing significant issues when it runs back to back to back..., etc.. I've pasted the query below and I'm looking to see if there is any expert out there that might know of a better way to structure this query so that it runs in a fraction of a second as opposed to the 2+ seconds it takes now.
SELECT acl_actions.id,
acl_actions.NAME,
acl_actions.category,
acl_actions.acltype,
acl_actions.aclaccess,
tt.access_override
FROM acl_actions
LEFT JOIN (SELECT acl_roles_users.user_id,
acl_roles_actions.action_id,
acl_roles_actions.access_override
FROM acl_roles_users
LEFT JOIN acl_roles_actions acl_roles_actions
ON acl_roles_actions.role_id =
acl_roles_users.role_id
AND acl_roles_actions.deleted = 0
WHERE ( acl_roles_users.user_id =
'XXXXXXXXX' )
AND ( acl_roles_users.deleted = 0 )) tt
ON tt.action_id = acl_actions.id
WHERE acl_actions.deleted = 0
ORDER BY acl_actions.category ASC,
acl_actions.NAME ASC;
Here's the results of EXPLAIN
+----+-------------+-------------------+------------+------+---------------------------------------------------------------+-----------------------+---------+------------------------------------------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------------+------------+------+---------------------------------------------------------------+-----------------------+---------+------------------------------------------+------+----------+--------------------------+
| 1 | SIMPLE | acl_actions | NULL | ref | idx_del_category_name_acltype_aclaccess,idx_del_category_name | idx_del_category_name | 2 | const | 3520 | 100.00 | Using index condition |
| 1 | SIMPLE | acl_roles_users | NULL | ref | idx_acluser_id | idx_acluser_id | 145 | const | 1 | 100.00 | Using where |
| 1 | SIMPLE | acl_roles_actions | NULL | ref | idx_acl_role_id,idx_del_override,idx_aclrole_action | idx_del_override | 147 | production.acl_roles_users.role_id,const | 312 | 100.00 | Using where; Using index |
+----+-------------+-------------------+------------+------+---------------------------------------------------------------+-----------------------+---------+------------------------------------------+------+----------+--------------------------+
CREATE TABLE
CREATE TABLE `acl_actions` (
`id` char(36) NOT NULL,
`date_entered` datetime DEFAULT NULL,
`date_modified` datetime DEFAULT NULL,
`modified_user_id` char(36) DEFAULT NULL,
`created_by` char(36) DEFAULT NULL,
`name` varchar(150) DEFAULT NULL,
`category` varchar(100) DEFAULT NULL,
`acltype` varchar(100) DEFAULT NULL,
`aclaccess` int(3) DEFAULT NULL,
`deleted` tinyint(1) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_aclaction_id_del` (`id`,`deleted`),
KEY `idx_category_name` (`category`,`name`),
KEY `idx_del_category_name_acltype_aclaccess` (`deleted`,`category`,`name`,`acltype`,`aclaccess`),
KEY `idx_del_category_name` (`deleted`,`category`,`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
CREATE TABLE `acl_roles_users` (
`id` char(36) NOT NULL,
`role_id` char(36) DEFAULT NULL,
`user_id` char(36) DEFAULT NULL,
`date_modified` datetime DEFAULT NULL,
`deleted` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `idx_aclrole_id` (`role_id`),
KEY `idx_acluser_id` (`user_id`),
KEY `idx_aclrole_user` (`role_id`,`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
CREATE TABLE `acl_roles_actions` (
`id` char(36) NOT NULL,
`role_id` char(36) DEFAULT NULL,
`action_id` char(36) DEFAULT NULL,
`access_override` int(3) DEFAULT NULL,
`date_modified` datetime DEFAULT NULL,
`deleted` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `idx_acl_role_id` (`role_id`),
KEY `idx_acl_action_id` (`action_id`),
KEY `idx_del_override` (`role_id`,`deleted`,`action_id`,`access_override`),
KEY `idx_aclrole_action` (`role_id`,`action_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

Optimizing a simple query with many left joins

I have quite a large query which is used for a user search on 'map_item'.
SELECT map_item_name
FROM map_item
LEFT JOIN map_section_item ON map_section_item_item_id = map_item_id
LEFT JOIN map_section ON map_section_id = map_section_item_section_id
LEFT JOIN map_item_flag ON map_item_flag_item_id = map_item_id
LEFT JOIN flag ON flag_id = map_item_flag_flag_id
LEFT JOIN map ON map_id = map_section_map_id
LEFT JOIN place_map ON place_map_map_id = map_id
LEFT JOIN place ON place_id = place_map_place_id
LEFT JOIN place_category ON place_category_place_id = place_id
LEFT JOIN category ON category_id = place_category_category_id
LEFT JOIN review ON review_map_item_id = map_item_id
LEFT JOIN map_price ON map_price_item_id = map_item_id
LEFT JOIN county_list ON place_address_county = county_id
'map_item' has 5399 records in total and none of the joined tables have much data in at all.
If I run this query without the left joins (SELECT map_item_name FROM map_item) it returns in 0.00s as expected, but the above query with the joins takes around 10.00s.
All of the left joins are required in the query due to the different filters that the user can apply to the search, however the original query was taking a long time to run (20 seconds or so), and after stripping out most parts of the query I was left with the above (which is just the left joins) and even this is taking 18 seconds to run.
Here is the explain statement from the query:
+----+-------------+-------------------+--------+----------------------------------+----------------------------------+---------+-----------------------------------------------------------+------+-----------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+--------+----------------------------------+----------------------------------+---------+-----------------------------------------------------------+------+-----------------------------------------------------------------+
| 1 | SIMPLE | map_item | ALL | NULL | NULL | NULL | NULL | 5455 | NULL |
| 1 | SIMPLE | map_section_item | index | NULL | map_section_item_section_id | 8 | NULL | 5330 | Using where; Using index; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | map_section | eq_ref | PRIMARY | PRIMARY | 4 | bestmeal.map_section_item.map_section_item_section_id | 1 | NULL |
| 1 | SIMPLE | map_item_flag | ALL | NULL | NULL | NULL | NULL | 1509 | Using where; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | flag | eq_ref | PRIMARY | PRIMARY | 4 | bestmeal.map_item_flag.map_item_flag_flag_id | 1 | Using index |
| 1 | SIMPLE | map | eq_ref | PRIMARY | PRIMARY | 4 | bestmeal.map_section.map_section_map_id | 1 | Using index |
| 1 | SIMPLE | place_map | index | NULL | branch_map_branch_id | 8 | NULL | 1275 | Using where; Using index; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | place | eq_ref | PRIMARY | PRIMARY | 4 | bestmeal.place_map.place_map_place_id | 1 | NULL |
| 1 | SIMPLE | place_category | ref | place_category_place_id | place_category_place_id | 4 | bestmeal.place.place_id | 1 | Using index |
| 1 | SIMPLE | category | eq_ref | PRIMARY | PRIMARY | 4 | bestmeal.place_category.place_category_category_id | 1 | Using index |
| 1 | SIMPLE | review | ref | review_map_item_id | review_map_item_id | 4 | bestmeal.map_item.map_item_id | 1 | Using index |
| 1 | SIMPLE | map_price | ref | map_price_item_id | map_price_item_id | 4 | bestmeal.map_item.map_item_id | 1 | Using index |
| 1 | SIMPLE | county_list | eq_ref | PRIMARY | PRIMARY | 4 | bestmeal.place.place_address_county | 1 | Using index |
+----+-------------+-------------------+--------+----------------------------------+----------------------------------+---------+-----------------------------------------------------------+------+-----------------------------------------------------------------+
All of these joins are made against indexed fields, and none of the tables that are joined have any unnecessary indexes in them which could be used instead of the intended index.
I'm not an expert when it comes to optimising queries, but I'm struggling to work out what I can do to speed this query up whilst keeping the left joins. I also can't really think of any alternative solutions which will return the same results without using the joins.
Does anybody have any ideas that will help me to increase the performance on this query or accomplish the user search using a different, faster method?
Edit
Table structures as requested:
CREATE TABLE `map_item` (
`map_item_id` int(11) NOT NULL AUTO_INCREMENT,
`map_item_account_id` int(11) NOT NULL DEFAULT '0',
`map_item_category_id` int(11) NOT NULL,
`map_item_name` varchar(255) DEFAULT NULL,
`map_item_description` text,
`map_item_tags` varchar(255) DEFAULT NULL,
`map_item_type` set('d','f') DEFAULT NULL,
PRIMARY KEY (`map_item_id`),
KEY `map_item_account_id` (`map_item_account_id`),
KEY `map_item_tags` (`map_item_tags`),
KEY `map_item_category_id` (`map_item_category_id`),
FULLTEXT KEY `map_item_keyword_search` (`map_item_name`,`map_item_description`,`map_item_tags`),
FULLTEXT KEY `map_item_name` (`map_item_name`),
FULLTEXT KEY `map_item_description` (`map_item_description`),
FULLTEXT KEY `map_item_tags_2` (`map_item_tags`)
) ENGINE=InnoDB AUTO_INCREMENT=5420 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `map_section_item` (
`map_section_item_id` int(11) NOT NULL AUTO_INCREMENT,
`map_section_item_section_id` int(11) NOT NULL DEFAULT '0',
`map_section_item_item_id` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`map_section_item_id`),
KEY `map_section_item_section_id` (`map_section_item_section_id`,`map_section_item_item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=24410 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `map_section` (
`map_section_id` int(11) NOT NULL AUTO_INCREMENT,
`map_section_map_id` int(11) NOT NULL DEFAULT '0',
`map_section_map_draft_id` int(11) NOT NULL DEFAULT '0',
`map_section_column` tinyint(1) NOT NULL DEFAULT '1',
`map_section_name` varchar(255) DEFAULT NULL,
`map_section_description` text,
PRIMARY KEY (`map_section_id`),
KEY `map_section_map_draft_id` (`map_section_map_draft_id`),
KEY `map_section_map_id` (`map_section_map_id`),
FULLTEXT KEY `index_name` (`map_section_name`)
) ENGINE=InnoDB AUTO_INCREMENT=4254 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `map_item_flag` (
`map_item_flag_id` int(11) NOT NULL AUTO_INCREMENT,
`map_item_flag_item_id` int(11) NOT NULL DEFAULT '0',
`map_item_flag_flag_id` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`map_item_flag_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1547 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `flag` (
`flag_id` int(11) NOT NULL AUTO_INCREMENT,
`flag_category_id` int(11) NOT NULL DEFAULT '0',
`flag_name` varchar(255) DEFAULT NULL,
`flag_description` varchar(255) DEFAULT NULL,
`flag_img` varchar(255) DEFAULT NULL,
`flag_order` tinyint(2) NOT NULL DEFAULT '0',
PRIMARY KEY (`flag_id`),
KEY `flag_category_id` (`flag_category_id`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `map` (
`map_id` int(11) NOT NULL AUTO_INCREMENT,
`map_account_id` int(11) NOT NULL DEFAULT '0',
`map_name` varchar(255) DEFAULT NULL,
`map_description` text,
`map_type` set('d','f') DEFAULT NULL,
`map_layout` set('columns','tabs','collapsed') DEFAULT NULL,
PRIMARY KEY (`map_id`),
KEY `map_account_id` (`map_account_id`)
) ENGINE=InnoDB AUTO_INCREMENT=138 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `place_map` (
`place_map_id` int(11) NOT NULL AUTO_INCREMENT,
`place_map_place_id` int(11) NOT NULL DEFAULT '0',
`place_map_map_id` int(11) NOT NULL DEFAULT '0',
`place_map_active` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`place_map_id`),
KEY `branch_map_branch_id` (`place_map_place_id`,`place_map_map_id`)
) ENGINE=InnoDB AUTO_INCREMENT=2176 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `place` (
`place_id` int(11) NOT NULL AUTO_INCREMENT,
`place_account_id` int(11) NOT NULL DEFAULT '0',
`place_name` varchar(120) DEFAULT NULL,
`place_alias` varchar(255) DEFAULT NULL,
`place_description` text,
`place_address_line_one` varchar(100) DEFAULT NULL,
`place_address_line_two` varchar(100) DEFAULT NULL,
`place_address_line_three` varchar(100) DEFAULT NULL,
`place_address_town` varchar(100) DEFAULT NULL,
`place_address_county` int(11) NOT NULL DEFAULT '0',
`place_address_postcode` varchar(10) DEFAULT NULL,
`place_address_latitude` decimal(11,8) DEFAULT NULL,
`place_address_longitude` decimal(11,8) DEFAULT NULL,
`place_phone` varchar(20) DEFAULT NULL,
`place_email` varchar(255) DEFAULT NULL,
`place_website` varchar(120) DEFAULT NULL,
`place_flag_initial_email` tinyint(1) NOT NULL DEFAULT '0',
`place_audit_admin_id` int(11) NOT NULL DEFAULT '0',
`place_last_audit_datetime` datetime DEFAULT NULL,
`place_created_by_admin_id` int(11) NOT NULL DEFAULT '0',
`place_created` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`place_tried_google` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`place_id`),
KEY `place_account_id` (`place_account_id`),
KEY `place_address_county` (`place_address_county`),
KEY `place_alias` (`place_alias`),
KEY `place_audit_admin_id` (`place_audit_admin_id`),
KEY `place_created_by_admin_id` (`place_created_by_admin_id`),
FULLTEXT KEY `place_name` (`place_name`),
FULLTEXT KEY `place_keyword_search` (`place_name`,`place_address_town`),
FULLTEXT KEY `place_address_town` (`place_address_town`)
) ENGINE=InnoDB AUTO_INCREMENT=135167 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `place_category` (
`place_category_id` int(11) NOT NULL AUTO_INCREMENT,
`place_category_place_id` int(11) NOT NULL DEFAULT '0',
`place_category_category_id` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`place_category_id`),
UNIQUE KEY `place_category_place_id` (`place_category_place_id`,`place_category_category_id`)
) ENGINE=InnoDB AUTO_INCREMENT=208987 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `category` (
`category_id` int(11) NOT NULL AUTO_INCREMENT,
`category_name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`category_id`)
) ENGINE=InnoDB AUTO_INCREMENT=168 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `review` (
`review_id` int(11) NOT NULL AUTO_INCREMENT,
`review_user_id` int(11) NOT NULL DEFAULT '0',
`review_place_id` int(11) NOT NULL DEFAULT '0',
`review_map_item_id` int(11) NOT NULL DEFAULT '0',
`review_otm_item_name` varchar(156) DEFAULT NULL,
`review_headline` varchar(255) DEFAULT NULL,
`review_message` text,
`review_rating` tinyint(1) NOT NULL DEFAULT '0',
`review_datetime` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`review_edited_datetime` datetime DEFAULT NULL,
`review_hidden` tinyint(1) NOT NULL DEFAULT '0',
`review_deleted` tinyint(1) NOT NULL DEFAULT '0',
`review_status` set('pending','published','hidden','deleted') NOT NULL,
PRIMARY KEY (`review_id`),
KEY `review_map_item_id` (`review_map_item_id`),
KEY `review_place_id` (`review_place_id`),
KEY `review_user_id` (`review_user_id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `map_price` (
`map_price_id` int(11) NOT NULL AUTO_INCREMENT,
`map_price_item_id` int(11) NOT NULL DEFAULT '0',
`map_price_label` varchar(50) DEFAULT NULL,
`map_price_value` decimal(10,2) DEFAULT NULL,
PRIMARY KEY (`map_price_id`),
KEY `map_price_item_id` (`map_price_item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=5872 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
CREATE TABLE `county_list` (
`county_id` int(11) NOT NULL AUTO_INCREMENT,
`county_country_id` int(11) NOT NULL DEFAULT '0',
`county_name` varchar(120) DEFAULT NULL,
`county_alias` varchar(120) DEFAULT NULL,
PRIMARY KEY (`county_id`),
KEY `county_alias` (`county_alias`),
KEY `county_country_id` (`county_country_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142 DEFAULT CHARSET=latin1 ROW_FORMAT=COMPACT
Look at these lines:
LEFT JOIN map_section_item ON map_section_item_item_id = map_item_id
| 1 | SIMPLE | map_section_item | index | NULL | map_section_item_section_id | 8 | NULL | 5330 | Using where; Using index; Using join buffer (Block Nested Loop) |
|
Notice "5330". That means it had to search about 5330 items to find the row it needed.
With a simple INDEX(map_section_item_item_id), it would go directly to the one (or few) row it needed. This would make the query run a lot faster.
Repeat for each other JOIN, at least for those with a "Rows" > 1.
Why LEFT? Is each "right" table optionally missing data?
A side issue: Don't prefix everything with the table name; it is too much clutter.
For MySQL, try using the STRAIGHT_JOIN clause...
SELECT STRAIGHT_JOIN map_item_name
FROM map_item
LEFT JOIN ...
STRAIGHT_JOIN tells MySQL to do the query in the order I've listed. This way it forces the map_item as the primary table and all the rest as lookup secondary tables...

How to optimize MySQL query with large data on left join?

The query below returns a set of User and for each row a number of relations from the user perpective who is searching (id = 4)
SELECT `users`.`firstname` AS firstname,
`users`.`lastname` AS lastname,
COUNT(`trusted_users`.`id`) AS number_of_friend_in_common,
CASE ... AS friend,
CASE ... AS facebook_invitable,
CASE ... AS address_book_invitable,
CASE ... AS virtual_user,
FROM `users`
LEFT OUTER JOIN `trusted_users`
ON `trusted_users`.`user_id` = 4 AND `trusted_users`.`trust_user_id` = `users`.`id`
LEFT OUTER JOIN `facebook_friends`
ON (`facebook_friends`.`user_id` = 4 AND `facebook_friends`.`friend_user_id` = `users`.`id`
OR `facebook_friends`.`user_id` = `users`.`id` AND `facebook_friends`.`friend_user_id` = 4)
LEFT OUTER JOIN `address_book_contacts`
ON `address_book_contacts`.`owner_id` = 4 AND `address_book_contacts`.`email_digest` = `users`.`email_digest`
LEFT OUTER JOIN `friends`
ON (`friends`.`me_id` = `users`.`id` AND `friends`.`him_id` = 4
OR `friends`.`me_id` = 4 AND `friends`.`him_id` = `users`.`id`)
WHERE `users`.`id` NOT IN
(SELECT CASE
WHEN `friends`.`me_id` = 4 THEN `friends`.`him_id`
ELSE `friends`.`me_id`
END
FROM `friends`
WHERE (`friends`.`status` = 0
AND `friends`.`him_id` = 4
AND `friends`.`him_status` = 7
OR `friends`.`status` = 0
AND `friends`.`me_id` = 4
AND `friends`.`me_status` = 7))
AND (`users`.`firstname` LIKE '%a%' OR `users`.`lastname` LIKE '%a%')
GROUP BY `users`.`id`
ORDER BY friend DESC,
facebook_invitable DESC,
address_book_invitable DESC,
number_of_friend_in_common DESC,
virtual_user DESC,
firstname,
lastname LIMIT 0, 20
Number of row for each table:
trusted_users: 255k
facebook_friends: 1k
address_book_contacts: 1.5M
friends: 70k
users: 32k
All fields of join are indexed. The query takes 1.1s which is not acceptable for the amount of data we have.
What am I doing wrong? Should I split in multiple query and paginate my self?
Edit 1: EXPLAIN result
+----+-------------+-----------------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+---------+----------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+---------+----------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+
| 1 | PRIMARY | users | ALL | PRIMARY,index_users_on_chat_id,index_users_login_facebook_id,index_users_on_login,index_users_on_parent_id,index_users_account_type,index_users_email_digest,index_users_id | NULL | NULL | NULL | 31847 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | trusted_users | ref | index_trusted_users_user,index_trusted_users_trust_user | index_trusted_users_trust_user | 5 | messenger_dev.users.id | 6 | Using where |
| 1 | PRIMARY | facebook_friends | index_merge | index_facebook_friends_user,index_facebook_friends_friend | index_facebook_friends_user,index_facebook_friends_friend | 5,5 | NULL | 2 | Using union(index_facebook_friends_user,index_facebook_friends_friend); Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | address_book_contacts | ref | index_address_book_contacts_owner_id,index_address_book_contacts_email | index_address_book_contacts_email | 767 | messenger_dev.users.email_digest | 1 | Using where |
| 1 | PRIMARY | friends | index_merge | index_friends_me_him,index_friends_me,index_friends_him | index_friends_him,index_friends_me | 5,5 | NULL | 18 | Using union(index_friends_him,index_friends_me); Using where; Using join buffer (Block Nested Loop) |
| 2 | SUBQUERY | friends | index_merge | index_friends_me_him,index_friends_me,index_friends_him | index_friends_him,index_friends_me | 5,5 | NULL | 18 | Using union(index_friends_him,index_friends_me); Using where |
+----+-------------+-----------------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+---------+----------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0,00 sec)
Edit 2: Table structure
CREATE TABLE `address_book_contacts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`email_digest` varchar(191) DEFAULT NULL,
`code` varchar(191) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`owner_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_address_book_contacts_owner_id` (`owner_id`),
KEY `index_address_book_contacts_email` (`email_digest`)
) ENGINE=InnoDB AUTO_INCREMENT=1598109 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `trusted_users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`friend_id` int(11) DEFAULT NULL,
`user_id` int(11) DEFAULT NULL,
`trust_user_id` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `index_trusted_users_on_friend_id` (`friend_id`),
KEY `index_trusted_users_user` (`user_id`),
KEY `index_trusted_users_trust_user` (`trust_user_id`),
CONSTRAINT `fk_rails_007c31c802` FOREIGN KEY (`trust_user_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_ca24cb4e23` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=275576 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `facebook_friends` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) DEFAULT NULL,
`friend_user_id` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`code` varchar(191) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_facebook_friends_user` (`user_id`),
KEY `index_facebook_friends_friend` (`friend_user_id`),
KEY `index_facebook_friends_code` (`code`(5)),
CONSTRAINT `fk_rails_78285a074e` FOREIGN KEY (`friend_user_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_aa3ac53a81` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1149 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `friends` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`me_id` int(11) DEFAULT NULL,
`him_id` int(11) DEFAULT NULL,
`owner_id` int(11) DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`me_status` int(11) DEFAULT '0',
`him_status` int(11) DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `index_friends_me_him` (`me_id`,`him_id`),
KEY `index_friends_me` (`me_id`),
KEY `index_friends_him` (`him_id`),
KEY `index_friends_owner` (`owner_id`),
CONSTRAINT `fk_rails_9fa3474d31` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_d3ebb6657f` FOREIGN KEY (`him_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_fccfd1b821` FOREIGN KEY (`me_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=95724 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`account_type` varchar(191) NOT NULL,
`firstname` varchar(191) DEFAULT NULL,
`lastname` varchar(191) DEFAULT NULL,
`login` varchar(191) DEFAULT NULL,
`avatar` varchar(191) DEFAULT NULL,
`gender` varchar(1) DEFAULT NULL,
`locale` varchar(191) DEFAULT NULL,
`birthdate` date DEFAULT NULL,
`password_digest` varchar(191) DEFAULT NULL,
`email_digest` varchar(191) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `index_users_on_login` (`login`),
KEY `index_users_account_type` (`account_type`),
KEY `index_users_email_digest` (`email_digest`),
KEY `index_uses_firstname` (`firstname`),
KEY `index_users_lastname` (`lastname`)
) ENGINE=InnoDB AUTO_INCREMENT=32516 DEFAULT CHARSET=utf8mb4;
It looks like MySQL is not choosing any of the available indicies for the users table.
First, run ANALYZE TABLE users;, then re-run the EXPLAIN command. Is the value of the first rows cell now substantially lower than 31847? If so, your problem should be solved!
If not, run OPTIMIZE TABLE users;, then re-run the EXPLAIN command. Is the value of the first rows cell now substantially lower than 31847? If so, your problem should be solved!
If neither of those steps help, try adding USE INDEX (PRIMARY) or USE INDEX (users_id) immediately after the FROM users portion of your query.
Hope this helps!