Why is MySQL query using join buffer? - mysql

The following query is using the join buffer and I was wondering if someone could explain to me why this is so. Just trying to gain more understanding about mysql and indexing.
mysql> EXPLAIN SELECT events.event_topic_id, event_topic_name, event_topic_image, event_type_name,city_name FROM events
-> JOIN event_topic ON event_topic.event_topic_id=events.event_topic_id
-> JOIN event_type ON event_type.event_type_id = event_topic.event_type_id
-> JOIN locations ON locations.location_id=events.location_id
-> JOIN city ON city.city_id=locations.city_id
-> WHERE event_date > NOW()
-> GROUP BY events.event_topic_id, city.city_id;
+----+-------------+-------------+--------+---------------------------------------+-----------------+---------+--------------------------------------+------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------+--------+---------------------------------------+-----------------+---------+--------------------------------------+------+----------+----------------------------------------------+
| 1 | SIMPLE | city | index | PRIMARY | city_name | 52 | NULL | 6 | 100.00 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | locations | ref | PRIMARY,city_id | city_id | 1 | PremiumCONNECT.city.city_id | 1 | 100.00 | Using index |
| 1 | SIMPLE | events | ref | location_id,event_topic_id,event_date | location_id | 2 | PremiumCONNECT.locations.location_id | 3 | 100.00 | Using where |
| 1 | SIMPLE | event_type | index | PRIMARY | event_type_name | 52 | NULL | 2 | 100.00 | Using index; Using join buffer |
| 1 | SIMPLE | event_topic | eq_ref | PRIMARY,event_type_id | PRIMARY | 1 | PremiumCONNECT.events.event_topic_id | 1 | 100.00 | Using where |
+----+-------------+-------------+--------+---------------------------------------+-----------------+---------+--------------------------------------+------+----------+----------------------------------------------+
Events table:
CREATE TABLE `events` (
`event_id` smallint(8) unsigned NOT NULL AUTO_INCREMENT,
`location_id` smallint(3) unsigned NOT NULL,
`event_date` datetime NOT NULL,
`event_topic_id` tinyint(3) unsigned NOT NULL,
PRIMARY KEY (`event_id`),
KEY `location_id` (`location_id`),
KEY `event_topic_id` (`event_topic_id`),
KEY `event_date` (`event_date`),
CONSTRAINT `events_ibfk_2` FOREIGN KEY (`event_topic_id`) REFERENCES `event_topic` (`event_topic_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `events_ibfk_3` FOREIGN KEY (`location_id`) REFERENCES `locations` (`location_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=91 DEFAULT CHARSET=latin1
Event topic table:
CREATE TABLE `event_topic` (
`event_topic_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`event_topic_name` varchar(100) DEFAULT NULL,
`event_topic_description` text NOT NULL,
`event_topic_cost` decimal(7,2) DEFAULT NULL,
`event_type_id` tinyint(3) unsigned NOT NULL,
`event_topic_clickthrough` tinytext,
`event_topic_length` varchar(6) NOT NULL,
`event_topic_image` varchar(41) DEFAULT NULL,
`event_topic_image_md5` char(32) NOT NULL,
PRIMARY KEY (`event_topic_id`),
KEY `event_type_id` (`event_type_id`),
KEY `topic_image_sha1` (`event_topic_image_md5`),
CONSTRAINT `event_topic_ibfk_1` FOREIGN KEY (`event_type_id`) REFERENCES `event_type` (`event_type_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1
Event type table:
CREATE TABLE `event_type` (
`event_type_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`event_type_name` varchar(50) NOT NULL,
`conf_email` text,
PRIMARY KEY (`event_type_id`),
KEY `event_type_name` (`event_type_name`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1
Locations table:
CREATE TABLE `locations` (
`location_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`location_name` varchar(50) NOT NULL,
`location_address` tinytext NOT NULL,
`location_capacity` smallint(6) NOT NULL,
`city_id` tinyint(3) unsigned NOT NULL,
`gps_coords` varchar(30) DEFAULT NULL,
PRIMARY KEY (`location_id`),
KEY `city_id` (`city_id`),
CONSTRAINT `locations_ibfk_1` FOREIGN KEY (`city_id`) REFERENCES `city` (`city_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=latin1
Cities table:
CREATE TABLE `city` (
`city_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`city_name` varchar(50) NOT NULL,
PRIMARY KEY (`city_id`),
UNIQUE KEY `city_name` (`city_name`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1

As it says in 'http://dev.mysql.com/doc/refman/5.1/en/explain-output.html': "Tables from earlier joins are read in portions into the join buffer, and then their rows are used from the buffer to perform the join with the current table."
So in your case, you had already joined event_topic, so the optimizer was able to use event_topic content from the join buffer.
Using a buffer is a good thing; you probably noticed the undesirable "Using temporary; Using filesort" on the first line of EXPLAIN output, which is probably from the GROUP BY and is probably unavoidable in this case.
By the way, will you run into problems with the "UNIQUE" constraint on city_name? I'm thinking of Springfield (two in New Jersey), Washington, Greenville, etc.

Try using:
"STRAIGHT_JOIN" and "FORCE INDEX":
EXPLAIN SELECT events.event_topic_id, event_topic_name, event_topic_image, event_type_name,city_name FROM events
-> straight_join event_topic force index(primary) ON event_topic.event_topic_id=events.event_topic_id
-> straight_join event_type force index(primary) ON event_type.event_type_id = event_topic.event_type_id
-> straight_join locations force index(primary) ON locations.location_id=events.location_id
-> straight_join city force index(primary) ON city.city_id=locations.city_id
-> WHERE event_date > NOW()
-> GROUP BY events.event_topic_id, city.city_id;
BTW, using a join buffer is not good. It means that you need to improve or reference to correct index.

Related

Row Constructor Expression selects not using index

we have this table
CREATE TABLE `resource_grant` (
`resource_grant_id` int(11) NOT NULL AUTO_INCREMENT,
`member_type_id` int(11) NOT NULL,
`member_ref` varchar(36) NOT NULL,
`resource_type_id` int(11) NOT NULL,
`resource_ref` varchar(36) NOT NULL,
`role_id` int(11) NOT NULL,
`modified_by` varchar(255) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
`parent_grant_id` int(11) DEFAULT NULL,
PRIMARY KEY (`resource_grant_id`),
UNIQUE KEY `member_ref` (`member_ref`,`member_type_id`,`resource_type_id`,`resource_ref`),
KEY `member_type_id` (`member_type_id`),
KEY `resource_type_id` (`resource_type_id`),
KEY `role_id` (`role_id`),
KEY `resource_ref` (`resource_ref`,`resource_type_id`),
KEY `idx_rg_parent_grant_id` (`parent_grant_id`),
KEY `resource_ref_2` (`resource_ref`,`member_ref`,`resource_type_id`,`member_type_id`,`role_id`),
CONSTRAINT `resource_grant_ibfk_1` FOREIGN KEY (`member_type_id`) REFERENCES `member_type` (`member_type_id`),
CONSTRAINT `resource_grant_ibfk_2` FOREIGN KEY (`resource_type_id`) REFERENCES `resource_type` (`resource_type_id`),
CONSTRAINT `resource_grant_ibfk_3` FOREIGN KEY (`role_id`) REFERENCES `role` (`role_id`)
) ENGINE=InnoDB;
and these related tables
CREATE TABLE `member_type` (
`member_type_id` int(11) NOT NULL AUTO_INCREMENT,
`member_type` varchar(36) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`member_type_id`),
UNIQUE KEY `member_type` (`member_type`),
KEY `member_type_2` (`member_type`)
) ENGINE=InnoDB;
CREATE TABLE `resource_type` (
`resource_type_id` int(11) NOT NULL AUTO_INCREMENT,
`resource_type` varchar(36) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`resource_type_id`),
UNIQUE KEY `resource_type` (`resource_type`),
KEY `resource_type_2` (`resource_type`)
) ENGINE=InnoDB;
CREATE TABLE `role` (
`role_id` int(11) NOT NULL AUTO_INCREMENT,
`role_ref` varchar(50) NOT NULL,
`name` varchar(256) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`role_id`),
UNIQUE KEY `role_ref` (`role_ref`),
KEY `role_ref_2` (`role_ref`)
) ENGINE=InnoDB;
and we need to run selects like these ("Row Constructor Expression" syntax) (basically "bulk selects")
SELECT rg.resource_grant_id
FROM resource_grant rg
JOIN resource_type rt ON rg.resource_type_id = rt.resource_type_id
JOIN member_type mt ON rg.member_type_id = mt.member_type_id
JOIN role r ON r.role_id = rg.role_id
WHERE
(rg.resource_ref, rg.member_ref, rt.resource_type, mt.member_type, r.role_ref)
IN
(
('759','624962','property','epc-user','role.171'),
('11974','624962','property','epc-user','role.171')
);
the selects take ~60s to run, which is unacceptably long
note that there IS an index for (resource_ref,member_ref,resource_type_id,member_type_id,role_id)
we also don't want to run n individual select statements - we need these "bulk selects".
mysql 5.6 docs talk about this style of select not using indexes but you can make it using some tricks
https://dev.mysql.com/doc/refman/5.6/en/row-constructor-optimization.html
https://dev.mysql.com/doc/refman/5.6/en/range-optimization.html
not sure what's missing for us in order to make it use the indexes
EDIT here's the plan
mysql> explain SELECT rg.resource_grant_id FROM resource_grant rg JOIN resource_type rt ON rg.resource_type_id = rt.resource_type_id JOIN member_type mt ON rg.member_type_id = mt.member_type_id JOIN role r ON r.role_id = rg.role_id WHERE (rg.resource_ref, rg.member_ref, rt.resource_type, mt.member_type, r.role_ref) IN ( ('759','624962','property','epc-user','role.171'), ('11974','624962','property','epc-user','role.171') );
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
| 1 | SIMPLE | rt | index | PRIMARY | resource_type | 38 | NULL | 3 | Using index |
| 1 | SIMPLE | mt | index | PRIMARY | member_type | 38 | NULL | 6 | Using index; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | rg | ref | member_type_id,resource_type_id,role_id | member_type_id | 4 | samsDB.mt.member_type_id | 2370237 | Using where |
| 1 | SIMPLE | r | eq_ref | PRIMARY | PRIMARY | 4 | samsDB.rg.role_id | 1 | Using where |
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
4 rows in set (0.53 sec)
Start by changing the where clause to:
WHERE rg.member_ref = '624962' AND
rt.resource_type = 'property' AND
mt.member_type = 'epc-user' AND
r.role_ref = 'role.171' AND
rg.resource_ref IN ('759', '11974')
The existing indexes are not quite optimal for this. You need an index where the first two keys are (member_ref, resource_ref) -- well, except in the most recent versions of MySQL which implement skip-scan index optimizations.
You might be able to change resource_ref_2 to:
KEY `resource_ref_2` (`member_ref`, `resource_ref`, `resource_type_id`, `member_type_id`, `role_id`),
I'm not surprised at 60s on 5.6. "Row constructors" have existed for a long time. But they were not optimized before 5.7.
Either upgrade or rewrite the WHERE as Gordon suggests.

How to speed up MYSQL query with multiple joins?

looking for help to speed up this query:
SELECT tickets.ticketid, datenew, products.name
FROM tickets
INNER JOIN ticketlines ON tickets.id = ticketlines.ticket
INNER JOIN products ON ticketlines.product = products.id
INNER JOIN receipts ON receipts.id = tickets.id
WHERE (category !='feb765ef-c8a8-4fa2-969c-90f67fe6b3be' AND category!='888f4893-f300-43b5-9933-d549ade744e0' AND category !='8f2031e8-64a4-4abf-8175-3d2bedd9f950' AND category !='ca370ced-1c3b-434c-905e-ec1bc709543b' AND category !='f92ff0ac-fa11-4a5f-a3dd-e0d9ed9c171a' AND category !='445e8605-1cd9-4714-b3fd-7389ac29c206' and category !='05143c54-8a7e-4ce2-97cc-f84f9cf41395' AND category !='8c78afea-b9e2-44cf-b497-c384045b3202' AND category !='95919f7f-ff2e-4aa1-8110-ef63c022c01b' AND category !='f4f88b05-38a1-4956-9182-4c04a0808df7') AND datedone IS NULL
ORDER BY ticketid
This is from a cash register database that I am using to pull data for order display. The structure is the receipt has the primary id, the timein and timedone(DATEDONE) timestamps. The ticket table has ticket id which is the same as receipt id. Ticketlines table is the line items on the receipt/ticket. And then Products table has the human readable definitions of the products.
The query is to pull all items that aren't completed, i.e. DATEDONE is null, and display the items that were ordered.
Ticket and Receipt tables have 15K rows, Ticketlines has ~20K rows, Products has 1.5k.
Pretty small data. But this query takes over 20 seconds. I think since I am using primary key IDs for everything, I don't need to index anything, but I am a total noob, so I'm happy to be told I'm wrong. I'll appreciate any help and can provide any further details. Thanks!
EDIT:
per comments, I am showing table structures. I'm sorry for the formatting nightmare, I'm unfamiliar with this platform and unsure how to make it more readable :-(
| receipts | CREATE TABLE `receipts` (
`ID` varchar(255) NOT NULL,
`MONEY` varchar(255) NOT NULL,
`DATENEW` datetime NOT NULL,
`ATTRIBUTES` mediumblob,
`PERSON` varchar(255) DEFAULT NULL,
`DATEDONE` datetime DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `RECEIPTS_FK_MONEY` (`MONEY`),
KEY `RECEIPTS_INX_1` (`DATENEW`),
CONSTRAINT `RECEIPTS_FK_MONEY` FOREIGN KEY (`MONEY`) REFERENCES `closedcash` (`MONEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| tickets | CREATE TABLE `tickets` (
`ID` varchar(255) NOT NULL,
`TICKETTYPE` int(11) NOT NULL DEFAULT '0',
`TICKETID` int(11) NOT NULL,
`PERSON` varchar(255) NOT NULL,
`CUSTOMER` varchar(255) DEFAULT NULL,
`STATUS` int(11) NOT NULL DEFAULT '0',
`DONE` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `TICKETS_FK_2` (`PERSON`),
KEY `TICKETS_CUSTOMERS_FK` (`CUSTOMER`),
KEY `TICKETS_TICKETID` (`TICKETTYPE`,`TICKETID`),
CONSTRAINT `TICKETS_CUSTOMERS_FK` FOREIGN KEY (`CUSTOMER`) REFERENCES `customers` (`ID`),
CONSTRAINT `TICKETS_FK_2` FOREIGN KEY (`PERSON`) REFERENCES `people` (`ID`),
CONSTRAINT `TICKETS_FK_ID` FOREIGN KEY (`ID`) REFERENCES `receipts` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| ticketlines | CREATE TABLE `ticketlines` (
`TICKET` varchar(255) NOT NULL,
`LINE` int(11) NOT NULL,
`PRODUCT` varchar(255) DEFAULT NULL,
`ATTRIBUTESETINSTANCE_ID` varchar(255) DEFAULT NULL,
`UNITS` double NOT NULL,
`PRICE` double NOT NULL,
`TAXID` varchar(255) NOT NULL,
`ATTRIBUTES` mediumblob,
PRIMARY KEY (`TICKET`,`LINE`),
KEY `TICKETLINES_FK_2` (`PRODUCT`),
KEY `TICKETLINES_ATTSETINST` (`ATTRIBUTESETINSTANCE_ID`),
KEY `TICKETLINES_FK_3` (`TAXID`),
CONSTRAINT `TICKETLINES_ATTSETINST` FOREIGN KEY (`ATTRIBUTESETINSTANCE_ID`) REFERENCES `attributesetinstance` (`ID`),
CONSTRAINT `TICKETLINES_FK_2` FOREIGN KEY (`PRODUCT`) REFERENCES `products` (`ID`),
CONSTRAINT `TICKETLINES_FK_3` FOREIGN KEY (`TAXID`) REFERENCES `taxes` (`ID`),
CONSTRAINT `TICKETLINES_FK_TICKET` FOREIGN KEY (`TICKET`) REFERENCES `tickets` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| products | CREATE TABLE `products` (
`ID` varchar(255) NOT NULL,
`REFERENCE` varchar(255) NOT NULL,
`CODE` varchar(255) NOT NULL,
`CODETYPE` varchar(255) DEFAULT NULL,
`NAME` varchar(255) NOT NULL,
`PRICEBUY` double NOT NULL,
`PRICESELL` double NOT NULL,
`CATEGORY` varchar(255) NOT NULL,
`TAXCAT` varchar(255) NOT NULL,
`ATTRIBUTESET_ID` varchar(255) DEFAULT NULL,
`STOCKCOST` double DEFAULT NULL,
`STOCKVOLUME` double DEFAULT NULL,
`IMAGE` mediumblob,
`ISCOM` bit(1) NOT NULL DEFAULT b'0',
`ISSCALE` bit(1) NOT NULL DEFAULT b'0',
`ISKITCHEN` bit(1) NOT NULL DEFAULT b'0',
`PRINTKB` bit(1) NOT NULL DEFAULT b'0',
`SENDSTATUS` bit(1) NOT NULL DEFAULT b'0',
`ISSERVICE` bit(1) NOT NULL DEFAULT b'0',
`ATTRIBUTES` mediumblob,
`DISPLAY` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `PRODUCTS_INX_0` (`REFERENCE`),
UNIQUE KEY `PRODUCTS_INX_1` (`CODE`),
UNIQUE KEY `PRODUCTS_NAME_INX` (`NAME`),
KEY `PRODUCTS_FK_1` (`CATEGORY`),
KEY `PRODUCTS_TAXCAT_FK` (`TAXCAT`),
KEY `PRODUCTS_ATTRSET_FK` (`ATTRIBUTESET_ID`),
CONSTRAINT `PRODUCTS_ATTRSET_FK` FOREIGN KEY (`ATTRIBUTESET_ID`) REFERENCES `attributeset` (`ID`),
CONSTRAINT `PRODUCTS_FK_1` FOREIGN KEY (`CATEGORY`) REFERENCES `categories` (`ID`),
CONSTRAINT `PRODUCTS_TAXCAT_FK` FOREIGN KEY (`TAXCAT`) REFERENCES `taxcategories` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
Also, here is EXPLAIN output:
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------+------------+--------+--------------------------+---------+---------+------------------------------------+-------+----------+----------------------------------------------+
| 1 | SIMPLE | receipts | NULL | ALL | PRIMARY | NULL | NULL | NULL | 14624 | 10.00 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | tickets | NULL | eq_ref | PRIMARY | PRIMARY | 767 | receipts.ID | 1 | 100.00 | NULL |
| 1 | SIMPLE | ticketlines | NULL | ref | PRIMARY,TICKETLINES_FK_2 | PRIMARY | 767 | receipts.ID | 1 | 100.00 | Using where |
| 1 | SIMPLE | products | NULL | eq_ref | PRIMARY,PRODUCTS_FK_1 | PRIMARY | 767 | ticketlines.PRODUCT | 1 | 97.97 | Using where |
+----+-------------+-------------+------------+--------+--------------------------+---------+---------+------------------------------------+-------+----------+----------------------------------------------+
4 rows in set, 1 warning (0.04 sec)
Try changing data types of all columns to the minimal required ones.
Add indexes on columns in WHERE.
Add index on ticketlines.ticket.
I ended up deleted 10K rows from every linked table and it queries running in .2 seconds now. I guess absent any more advice, i will be doing annual cleanup of old tickets.

Fulltext working on MySQL 5.6, not 5.7

I have 2 main tables (professional_account & professional_profile) and 3 sub tables (prof_knows_about, prof_keywords & prof_recent_jobs) which are 1:many off professional_profile.
A FT search should match the search term in professional_profile.professional_statement, OR any row (for that professional_id) of prof_knows_about.knowledge_desc, OR any row of prof_keywords.keyword OR any row of prof_recent_jobs.job_title
In a 5.6.35 server it works as expected, but in 5.7.10 I only get results if the search term is in professional_profile.professional_statement or prof_knows_about.knowledge_desc - the search term is not found in the other two tables. Both servers run on freeBSD UNIX.
I tried both servers with the query pasted into the shell. Both also give same EXPLAIN output. Used a dump from the 5.7 on the 5.6 to ensure same data and table creates.
I did a show variables on both servers, but hard to compare as there are many extra values in the 5.7. Set both to same sql_mode.
Tables, query and explain listed below.
Is this an issue with my query, or my 5.7 server?
select email, PROFILE.*
FROM professional_account as ACC
join professional_profile as PROFILE on ACC.professional_id=PROFILE.professional_id
join prof_knows_about as KNOWS on PROFILE.professional_id=KNOWS.professional_id
join prof_keywords as KEYW on PROFILE.professional_id = KEYW.professional_id
join prof_recent_jobs as ROLES on PROFILE.professional_id=ROLES.professional_id
where status in ('LIVE','EDITS')
and (
match (knowledge_desc) against ('ramble')
or match (job_title) against ('ramble')
or match (keyword) against ('ramble')
or match (professional_statement) against ('ramble')
)
group by PROFILE.professional_profile_id
+----+-------------+---------+------------+--------+--------------------------------+-----------------+---------+---------------------------+------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+--------+--------------------------------+-----------------+---------+---------------------------+------+----------+---------------------------------+
| 1 | SIMPLE | KNOWS | NULL | ALL | professional_id | NULL | NULL | NULL | 43 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | PROFILE | NULL | ref | PRIMARY,professional_statement | professional_id | 4 | xxx.KNOWS.professional_id | 1 | 28.57 | Using where |
| 1 | SIMPLE | ACC | NULL | eq_ref | PRIMARY | PRIMARY | 4 | xxx.KNOWS.professional_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | ROLES | NULL | ref | professional_id | professional_id | 4 | xxx.KNOWS.professional_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | KEYW | NULL | ref | professional_id | professional_id | 4 | xxx.KNOWS.professional_id | 2 | 100.00 | Using where |
+----+-------------+---------+------------+--------+--------------------------------+-----------------+---------+---------------------------+------+----------+---------------------------------+
CREATE TABLE professional_account (
professional_id int(11) NOT NULL AUTO_INCREMENT,
email varchar(60) NOT NULL,
PRIMARY KEY (professional_id),
KEY email (email)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE professional_profile (
professional_profile_id int(11) NOT NULL AUTO_INCREMENT,
professional_id int(11) NOT NULL,
status enum('BLANK','PART_WAIT','PART_APP','COMP_WAIT','COMP_APP','LIVE','EDITS') DEFAULT NULL,
professional_statement text,
PRIMARY KEY (professional_profile_id),
KEY professional_id (professional_id),
FULLTEXT KEY professional_statement (professional_statement),
CONSTRAINT fk_professional_profile_professional_account FOREIGN KEY (professional_id) REFERENCES professional_account (professional_id) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE prof_knows_about (
prof_knows_about_id int(11) NOT NULL AUTO_INCREMENT,
professional_id int(11) NOT NULL,
knowledge_desc varchar(100) DEFAULT NULL,
PRIMARY KEY (prof_knows_about_id),
KEY professional_id (professional_id),
FULLTEXT KEY knowledge_desc (knowledge_desc),
CONSTRAINT fk_prof_knows_about_professional_profile1 FOREIGN KEY (professional_id) REFERENCES professional_profile (professional_id) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE prof_keywords (
prof_keywords_id int(11) NOT NULL AUTO_INCREMENT,
professional_id int(11) NOT NULL,
keyword varchar(20) DEFAULT NULL,
PRIMARY KEY (prof_keywords_id),
KEY professional_id (professional_id),
FULLTEXT KEY keyword (keyword),
CONSTRAINT fk_prof_keywords_professional_profile1 FOREIGN KEY (professional_id) REFERENCES professional_profile (professional_id) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE prof_recent_jobs (
prof_recent_jobs_id int(11) NOT NULL AUTO_INCREMENT,
professional_id int(11) NOT NULL,
job_title varchar(100) DEFAULT NULL,
experience tinyint(2) DEFAULT NULL,
PRIMARY KEY (prof_recent_jobs_id),
KEY professional_id (professional_id),
FULLTEXT KEY job_title (job_title),
CONSTRAINT fk_prof_recent_jobs_professional_profile1 FOREIGN KEY (professional_id) REFERENCES professional_profile (professional_id) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE prof_keywords (
prof_keywords_id int(11) NOT NULL AUTO_INCREMENT,
professional_id int(11) NOT NULL,
keyword varchar(20) DEFAULT NULL,
PRIMARY KEY (prof_keywords_id),
KEY professional_id (professional_id),
FULLTEXT KEY keyword (keyword),
CONSTRAINT fk_prof_keywords_professional_profile1 FOREIGN KEY (professional_id) REFERENCES professional_profile (professional_id) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

effective query on a table with over 1 million record

I am really struggling with the following two queries. I have two tables both with over a million records. The first query runs with 30 seconds and the second one runs with over 7 mins.
Basically I would like to get the count class.id and scan_layers.id based on the Lot. What would be the most effective way to do it?
select count(class.id), count(scan_layers.id), Lot, Verified from class left join scan_layers on (class.ScanLayer = scan_layers.id) group by Lot;
select count(class.id), count(distinct scan_layers.id), Lot, Verified from class left join scan_layers on (class.ScanLayer = scan_layers.id) group by Lot;
As I explain it, they both giving me the same explain.
+----+-------------+--------------------+--------+---------------+------------------------+---------+----------------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------+--------+---------------+------------------------+---------+----------------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | class | index | NULL | defects_scan_layers_fk | 4 | NULL | 4417159 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | scan_layers | eq_ref | PRIMARY | PRIMARY | 4 | cdb.class.ScanLayer | 1 | |
+----+-------------+--------------------+--------+---------------+------------------------+---------+----------------------------------+---------+----------------------------------------------+
CREATE TABLE `class` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`ScanLayer` int(10) unsigned NOT NULL,
`Type` enum('Regular','Critical') NOT NULL DEFAULT 'Regular',
PRIMARY KEY (`id`),
UNIQUE KEY `Defect_UNIQUE` (`ScanLayer`),
KEY `class_scan_layers_fk` (`ScanLayer`),
CONSTRAINT `class_scan_layers_fk` FOREIGN KEY (`ScanLayer`) REFERENCES `scan_layers` (`id`) ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `scan_layers` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`LayerInfo` int(10) unsigned NOT NULL,
`Lot` varchar(45) NOT NULL DEFAULT 'DEFAULT',
`PhysicalID` int(10) unsigned NOT NULL,
`Scanned` datetime DEFAULT NULL,
`ScannedMachine` int(10) unsigned DEFAULT NULL,
`DefectsCount` int(10) unsigned NOT NULL DEFAULT '0',
`MovesCount` int(10) unsigned NOT NULL DEFAULT '0',
`Verified` datetime DEFAULT NULL,
`VerifiedMachine` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ScanLayer_UNIQUE` (`LayerInfo`,`Lot`,`PhysicalID`),
KEY `scan_layers_layer_infos_fk` (`LayerInfo`),
KEY `scan_layers_scanned_machines_fk` (`ScannedMachine`),
KEY `scan_layers_verified_machines_fk` (`VerifiedMachine`),
KEY `scan_layers_verified` (`Verified`),
KEY `scan_layers_lot` (`Lot`),
CONSTRAINT `scan_layers_layer_infos_fk` FOREIGN KEY (`LayerInfo`) REFERENCES `layer_infos` (`id`) ON UPDATE CASCADE,
CONSTRAINT `scan_layers_scanned_machines_fk` FOREIGN KEY (`ScannedMachine`) REFERENCES `machines` (`id`) ON UPDATE CASCADE,
CONSTRAINT `scan_layers_verified_machines_fk` FOREIGN KEY (`VerifiedMachine`) REFERENCES `machines` (`id`) ON UPDATE CASCADE,
) ENGINE=InnoDB
Thanks a lot!

JOIN very slow when using RIGHT JOIN on this query

I'm having a problem with this query that takes several seconds to complete. I already tried many optimizations but I'm shooting blanks at this point.
The tables are the following (and are not absolutely normalized fully especially the tracks table)
CREATE TABLE `tracks` (
`id` int(14) unsigned NOT NULL AUTO_INCREMENT,
`artist` varchar(200) NOT NULL,
`track` varchar(200) NOT NULL,
`album` varchar(200) NOT NULL,
`path` text NOT NULL,
`tags` text NOT NULL,
`priority` int(10) NOT NULL DEFAULT '0',
`lastplayed` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`lastrequested` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`usable` int(1) NOT NULL DEFAULT '0',
`accepter` varchar(200) NOT NULL DEFAULT '',
`lasteditor` varchar(200) NOT NULL DEFAULT '',
`hash` varchar(40) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `hash` (`hash`),
FULLTEXT KEY `searchindex` (`tags`,`artist`,`track`,`album`),
FULLTEXT KEY `artist` (`artist`,`track`,`album`,`tags`)
) ENGINE=MyISAM AUTO_INCREMENT=3336 DEFAULT CHARSET=utf8
CREATE TABLE `esong` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`hash` varchar(40) COLLATE utf8_bin NOT NULL,
`len` int(10) unsigned NOT NULL,
`meta` text COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `hash` (`hash`)
) ENGINE=InnoDB AUTO_INCREMENT=16032 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
CREATE TABLE `efave` (
`id` int(10) unsigned NOT NULL DEFAULT '0',
`inick` int(10) unsigned NOT NULL,
`isong` int(10) unsigned NOT NULL,
UNIQUE KEY `inick` (`inick`,`isong`),
KEY `isong` (`isong`),
CONSTRAINT `inick` FOREIGN KEY (`inick`) REFERENCES `enick` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `isong` FOREIGN KEY (`isong`) REFERENCES `esong` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8
CREATE TABLE `enick` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT
`nick` varchar(30) COLLATE utf8_bin NOT NULL,
`dta` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`dtb` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
KEY `nick` (`nick`)
) ENGINE=InnoDB AUTO_INCREMENT=488 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
and the query I'm trying to execute with a normal speed is the following
SELECT esong.meta, tracks.id FROM tracks RIGHT JOIN esong ON tracks.hash = esong.hash JOIN efave ON efave.isong = esong.id JOIN enick ON efave.inick = enick.id WHERE enick.nick = lower('nickname');
Where if you remove the RIGHT JOIN and change it to JOIN it is fast
The EXPLAIN gives me this result, it seems there is a small problem in the efave selection but I have no idea how to get that out
+----+-------------+--------+--------+---------------+---------+---------+-----------------------+------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+--------+---------------+---------+---------+-----------------------+------+----------+--------------------------+
| 1 | SIMPLE | enick | ref | PRIMARY,nick | nick | 92 | const | 1 | 100.00 | Using where; Using index |
| 1 | SIMPLE | efave | ref | inick,isong | inick | 4 | radiosite.enick.id | 12 | 100.00 | Using index |
| 1 | SIMPLE | esong | eq_ref | PRIMARY | PRIMARY | 4 | radiosite.efave.isong | 1 | 100.00 | |
| 1 | SIMPLE | tracks | ALL | hash | NULL | NULL | NULL | 3210 | 100.00 | |
+----+-------------+--------+--------+---------------+---------+---------+-----------------------+------+----------+--------------------------+
Your explain looks clean, the only thing that stands out to me is the fact that the esong table is using a collate of utf8_bin, and the tracks table doesn't have a collation specified, which means it is probably using another collation type. Try aligning your collations and see how the join performs.
Have you checked your Execution Plan? If not, run your query to include it. Your Right Join may be doing an Index Scan instead of an Index Seek. Or you may be lacking indexes. Either way, you need to look at your Execution Plan so you can optimize your query better. No one will really be able to tell you how to make it faster using a Right Join (or a Join for that matter) until you know what the real problem is. Here are some links..
For MySQL: http://dev.mysql.com/doc/refman/5.5/en/execution-plan-information.html
For SqlServer: http://www.sql-server-performance.com/2006/query-execution-plan-analysis/