How to speed up MYSQL query with multiple joins? - mysql

looking for help to speed up this query:
SELECT tickets.ticketid, datenew, products.name
FROM tickets
INNER JOIN ticketlines ON tickets.id = ticketlines.ticket
INNER JOIN products ON ticketlines.product = products.id
INNER JOIN receipts ON receipts.id = tickets.id
WHERE (category !='feb765ef-c8a8-4fa2-969c-90f67fe6b3be' AND category!='888f4893-f300-43b5-9933-d549ade744e0' AND category !='8f2031e8-64a4-4abf-8175-3d2bedd9f950' AND category !='ca370ced-1c3b-434c-905e-ec1bc709543b' AND category !='f92ff0ac-fa11-4a5f-a3dd-e0d9ed9c171a' AND category !='445e8605-1cd9-4714-b3fd-7389ac29c206' and category !='05143c54-8a7e-4ce2-97cc-f84f9cf41395' AND category !='8c78afea-b9e2-44cf-b497-c384045b3202' AND category !='95919f7f-ff2e-4aa1-8110-ef63c022c01b' AND category !='f4f88b05-38a1-4956-9182-4c04a0808df7') AND datedone IS NULL
ORDER BY ticketid
This is from a cash register database that I am using to pull data for order display. The structure is the receipt has the primary id, the timein and timedone(DATEDONE) timestamps. The ticket table has ticket id which is the same as receipt id. Ticketlines table is the line items on the receipt/ticket. And then Products table has the human readable definitions of the products.
The query is to pull all items that aren't completed, i.e. DATEDONE is null, and display the items that were ordered.
Ticket and Receipt tables have 15K rows, Ticketlines has ~20K rows, Products has 1.5k.
Pretty small data. But this query takes over 20 seconds. I think since I am using primary key IDs for everything, I don't need to index anything, but I am a total noob, so I'm happy to be told I'm wrong. I'll appreciate any help and can provide any further details. Thanks!
EDIT:
per comments, I am showing table structures. I'm sorry for the formatting nightmare, I'm unfamiliar with this platform and unsure how to make it more readable :-(
| receipts | CREATE TABLE `receipts` (
`ID` varchar(255) NOT NULL,
`MONEY` varchar(255) NOT NULL,
`DATENEW` datetime NOT NULL,
`ATTRIBUTES` mediumblob,
`PERSON` varchar(255) DEFAULT NULL,
`DATEDONE` datetime DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `RECEIPTS_FK_MONEY` (`MONEY`),
KEY `RECEIPTS_INX_1` (`DATENEW`),
CONSTRAINT `RECEIPTS_FK_MONEY` FOREIGN KEY (`MONEY`) REFERENCES `closedcash` (`MONEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| tickets | CREATE TABLE `tickets` (
`ID` varchar(255) NOT NULL,
`TICKETTYPE` int(11) NOT NULL DEFAULT '0',
`TICKETID` int(11) NOT NULL,
`PERSON` varchar(255) NOT NULL,
`CUSTOMER` varchar(255) DEFAULT NULL,
`STATUS` int(11) NOT NULL DEFAULT '0',
`DONE` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `TICKETS_FK_2` (`PERSON`),
KEY `TICKETS_CUSTOMERS_FK` (`CUSTOMER`),
KEY `TICKETS_TICKETID` (`TICKETTYPE`,`TICKETID`),
CONSTRAINT `TICKETS_CUSTOMERS_FK` FOREIGN KEY (`CUSTOMER`) REFERENCES `customers` (`ID`),
CONSTRAINT `TICKETS_FK_2` FOREIGN KEY (`PERSON`) REFERENCES `people` (`ID`),
CONSTRAINT `TICKETS_FK_ID` FOREIGN KEY (`ID`) REFERENCES `receipts` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| ticketlines | CREATE TABLE `ticketlines` (
`TICKET` varchar(255) NOT NULL,
`LINE` int(11) NOT NULL,
`PRODUCT` varchar(255) DEFAULT NULL,
`ATTRIBUTESETINSTANCE_ID` varchar(255) DEFAULT NULL,
`UNITS` double NOT NULL,
`PRICE` double NOT NULL,
`TAXID` varchar(255) NOT NULL,
`ATTRIBUTES` mediumblob,
PRIMARY KEY (`TICKET`,`LINE`),
KEY `TICKETLINES_FK_2` (`PRODUCT`),
KEY `TICKETLINES_ATTSETINST` (`ATTRIBUTESETINSTANCE_ID`),
KEY `TICKETLINES_FK_3` (`TAXID`),
CONSTRAINT `TICKETLINES_ATTSETINST` FOREIGN KEY (`ATTRIBUTESETINSTANCE_ID`) REFERENCES `attributesetinstance` (`ID`),
CONSTRAINT `TICKETLINES_FK_2` FOREIGN KEY (`PRODUCT`) REFERENCES `products` (`ID`),
CONSTRAINT `TICKETLINES_FK_3` FOREIGN KEY (`TAXID`) REFERENCES `taxes` (`ID`),
CONSTRAINT `TICKETLINES_FK_TICKET` FOREIGN KEY (`TICKET`) REFERENCES `tickets` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| products | CREATE TABLE `products` (
`ID` varchar(255) NOT NULL,
`REFERENCE` varchar(255) NOT NULL,
`CODE` varchar(255) NOT NULL,
`CODETYPE` varchar(255) DEFAULT NULL,
`NAME` varchar(255) NOT NULL,
`PRICEBUY` double NOT NULL,
`PRICESELL` double NOT NULL,
`CATEGORY` varchar(255) NOT NULL,
`TAXCAT` varchar(255) NOT NULL,
`ATTRIBUTESET_ID` varchar(255) DEFAULT NULL,
`STOCKCOST` double DEFAULT NULL,
`STOCKVOLUME` double DEFAULT NULL,
`IMAGE` mediumblob,
`ISCOM` bit(1) NOT NULL DEFAULT b'0',
`ISSCALE` bit(1) NOT NULL DEFAULT b'0',
`ISKITCHEN` bit(1) NOT NULL DEFAULT b'0',
`PRINTKB` bit(1) NOT NULL DEFAULT b'0',
`SENDSTATUS` bit(1) NOT NULL DEFAULT b'0',
`ISSERVICE` bit(1) NOT NULL DEFAULT b'0',
`ATTRIBUTES` mediumblob,
`DISPLAY` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `PRODUCTS_INX_0` (`REFERENCE`),
UNIQUE KEY `PRODUCTS_INX_1` (`CODE`),
UNIQUE KEY `PRODUCTS_NAME_INX` (`NAME`),
KEY `PRODUCTS_FK_1` (`CATEGORY`),
KEY `PRODUCTS_TAXCAT_FK` (`TAXCAT`),
KEY `PRODUCTS_ATTRSET_FK` (`ATTRIBUTESET_ID`),
CONSTRAINT `PRODUCTS_ATTRSET_FK` FOREIGN KEY (`ATTRIBUTESET_ID`) REFERENCES `attributeset` (`ID`),
CONSTRAINT `PRODUCTS_FK_1` FOREIGN KEY (`CATEGORY`) REFERENCES `categories` (`ID`),
CONSTRAINT `PRODUCTS_TAXCAT_FK` FOREIGN KEY (`TAXCAT`) REFERENCES `taxcategories` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
Also, here is EXPLAIN output:
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------+------------+--------+--------------------------+---------+---------+------------------------------------+-------+----------+----------------------------------------------+
| 1 | SIMPLE | receipts | NULL | ALL | PRIMARY | NULL | NULL | NULL | 14624 | 10.00 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | tickets | NULL | eq_ref | PRIMARY | PRIMARY | 767 | receipts.ID | 1 | 100.00 | NULL |
| 1 | SIMPLE | ticketlines | NULL | ref | PRIMARY,TICKETLINES_FK_2 | PRIMARY | 767 | receipts.ID | 1 | 100.00 | Using where |
| 1 | SIMPLE | products | NULL | eq_ref | PRIMARY,PRODUCTS_FK_1 | PRIMARY | 767 | ticketlines.PRODUCT | 1 | 97.97 | Using where |
+----+-------------+-------------+------------+--------+--------------------------+---------+---------+------------------------------------+-------+----------+----------------------------------------------+
4 rows in set, 1 warning (0.04 sec)

Try changing data types of all columns to the minimal required ones.
Add indexes on columns in WHERE.
Add index on ticketlines.ticket.

I ended up deleted 10K rows from every linked table and it queries running in .2 seconds now. I guess absent any more advice, i will be doing annual cleanup of old tickets.

Related

Row Constructor Expression selects not using index

we have this table
CREATE TABLE `resource_grant` (
`resource_grant_id` int(11) NOT NULL AUTO_INCREMENT,
`member_type_id` int(11) NOT NULL,
`member_ref` varchar(36) NOT NULL,
`resource_type_id` int(11) NOT NULL,
`resource_ref` varchar(36) NOT NULL,
`role_id` int(11) NOT NULL,
`modified_by` varchar(255) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
`parent_grant_id` int(11) DEFAULT NULL,
PRIMARY KEY (`resource_grant_id`),
UNIQUE KEY `member_ref` (`member_ref`,`member_type_id`,`resource_type_id`,`resource_ref`),
KEY `member_type_id` (`member_type_id`),
KEY `resource_type_id` (`resource_type_id`),
KEY `role_id` (`role_id`),
KEY `resource_ref` (`resource_ref`,`resource_type_id`),
KEY `idx_rg_parent_grant_id` (`parent_grant_id`),
KEY `resource_ref_2` (`resource_ref`,`member_ref`,`resource_type_id`,`member_type_id`,`role_id`),
CONSTRAINT `resource_grant_ibfk_1` FOREIGN KEY (`member_type_id`) REFERENCES `member_type` (`member_type_id`),
CONSTRAINT `resource_grant_ibfk_2` FOREIGN KEY (`resource_type_id`) REFERENCES `resource_type` (`resource_type_id`),
CONSTRAINT `resource_grant_ibfk_3` FOREIGN KEY (`role_id`) REFERENCES `role` (`role_id`)
) ENGINE=InnoDB;
and these related tables
CREATE TABLE `member_type` (
`member_type_id` int(11) NOT NULL AUTO_INCREMENT,
`member_type` varchar(36) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`member_type_id`),
UNIQUE KEY `member_type` (`member_type`),
KEY `member_type_2` (`member_type`)
) ENGINE=InnoDB;
CREATE TABLE `resource_type` (
`resource_type_id` int(11) NOT NULL AUTO_INCREMENT,
`resource_type` varchar(36) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`resource_type_id`),
UNIQUE KEY `resource_type` (`resource_type`),
KEY `resource_type_2` (`resource_type`)
) ENGINE=InnoDB;
CREATE TABLE `role` (
`role_id` int(11) NOT NULL AUTO_INCREMENT,
`role_ref` varchar(50) NOT NULL,
`name` varchar(256) NOT NULL,
`modified_by` varchar(36) NOT NULL,
`modified_timestamp` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`role_id`),
UNIQUE KEY `role_ref` (`role_ref`),
KEY `role_ref_2` (`role_ref`)
) ENGINE=InnoDB;
and we need to run selects like these ("Row Constructor Expression" syntax) (basically "bulk selects")
SELECT rg.resource_grant_id
FROM resource_grant rg
JOIN resource_type rt ON rg.resource_type_id = rt.resource_type_id
JOIN member_type mt ON rg.member_type_id = mt.member_type_id
JOIN role r ON r.role_id = rg.role_id
WHERE
(rg.resource_ref, rg.member_ref, rt.resource_type, mt.member_type, r.role_ref)
IN
(
('759','624962','property','epc-user','role.171'),
('11974','624962','property','epc-user','role.171')
);
the selects take ~60s to run, which is unacceptably long
note that there IS an index for (resource_ref,member_ref,resource_type_id,member_type_id,role_id)
we also don't want to run n individual select statements - we need these "bulk selects".
mysql 5.6 docs talk about this style of select not using indexes but you can make it using some tricks
https://dev.mysql.com/doc/refman/5.6/en/row-constructor-optimization.html
https://dev.mysql.com/doc/refman/5.6/en/range-optimization.html
not sure what's missing for us in order to make it use the indexes
EDIT here's the plan
mysql> explain SELECT rg.resource_grant_id FROM resource_grant rg JOIN resource_type rt ON rg.resource_type_id = rt.resource_type_id JOIN member_type mt ON rg.member_type_id = mt.member_type_id JOIN role r ON r.role_id = rg.role_id WHERE (rg.resource_ref, rg.member_ref, rt.resource_type, mt.member_type, r.role_ref) IN ( ('759','624962','property','epc-user','role.171'), ('11974','624962','property','epc-user','role.171') );
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
| 1 | SIMPLE | rt | index | PRIMARY | resource_type | 38 | NULL | 3 | Using index |
| 1 | SIMPLE | mt | index | PRIMARY | member_type | 38 | NULL | 6 | Using index; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | rg | ref | member_type_id,resource_type_id,role_id | member_type_id | 4 | samsDB.mt.member_type_id | 2370237 | Using where |
| 1 | SIMPLE | r | eq_ref | PRIMARY | PRIMARY | 4 | samsDB.rg.role_id | 1 | Using where |
+----+-------------+-------+--------+-----------------------------------------+----------------+---------+--------------------------+---------+----------------------------------------------------+
4 rows in set (0.53 sec)
Start by changing the where clause to:
WHERE rg.member_ref = '624962' AND
rt.resource_type = 'property' AND
mt.member_type = 'epc-user' AND
r.role_ref = 'role.171' AND
rg.resource_ref IN ('759', '11974')
The existing indexes are not quite optimal for this. You need an index where the first two keys are (member_ref, resource_ref) -- well, except in the most recent versions of MySQL which implement skip-scan index optimizations.
You might be able to change resource_ref_2 to:
KEY `resource_ref_2` (`member_ref`, `resource_ref`, `resource_type_id`, `member_type_id`, `role_id`),
I'm not surprised at 60s on 5.6. "Row constructors" have existed for a long time. But they were not optimized before 5.7.
Either upgrade or rewrite the WHERE as Gordon suggests.

How to optimize MySQL query with large data on left join?

The query below returns a set of User and for each row a number of relations from the user perpective who is searching (id = 4)
SELECT `users`.`firstname` AS firstname,
`users`.`lastname` AS lastname,
COUNT(`trusted_users`.`id`) AS number_of_friend_in_common,
CASE ... AS friend,
CASE ... AS facebook_invitable,
CASE ... AS address_book_invitable,
CASE ... AS virtual_user,
FROM `users`
LEFT OUTER JOIN `trusted_users`
ON `trusted_users`.`user_id` = 4 AND `trusted_users`.`trust_user_id` = `users`.`id`
LEFT OUTER JOIN `facebook_friends`
ON (`facebook_friends`.`user_id` = 4 AND `facebook_friends`.`friend_user_id` = `users`.`id`
OR `facebook_friends`.`user_id` = `users`.`id` AND `facebook_friends`.`friend_user_id` = 4)
LEFT OUTER JOIN `address_book_contacts`
ON `address_book_contacts`.`owner_id` = 4 AND `address_book_contacts`.`email_digest` = `users`.`email_digest`
LEFT OUTER JOIN `friends`
ON (`friends`.`me_id` = `users`.`id` AND `friends`.`him_id` = 4
OR `friends`.`me_id` = 4 AND `friends`.`him_id` = `users`.`id`)
WHERE `users`.`id` NOT IN
(SELECT CASE
WHEN `friends`.`me_id` = 4 THEN `friends`.`him_id`
ELSE `friends`.`me_id`
END
FROM `friends`
WHERE (`friends`.`status` = 0
AND `friends`.`him_id` = 4
AND `friends`.`him_status` = 7
OR `friends`.`status` = 0
AND `friends`.`me_id` = 4
AND `friends`.`me_status` = 7))
AND (`users`.`firstname` LIKE '%a%' OR `users`.`lastname` LIKE '%a%')
GROUP BY `users`.`id`
ORDER BY friend DESC,
facebook_invitable DESC,
address_book_invitable DESC,
number_of_friend_in_common DESC,
virtual_user DESC,
firstname,
lastname LIMIT 0, 20
Number of row for each table:
trusted_users: 255k
facebook_friends: 1k
address_book_contacts: 1.5M
friends: 70k
users: 32k
All fields of join are indexed. The query takes 1.1s which is not acceptable for the amount of data we have.
What am I doing wrong? Should I split in multiple query and paginate my self?
Edit 1: EXPLAIN result
+----+-------------+-----------------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+---------+----------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+---------+----------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+
| 1 | PRIMARY | users | ALL | PRIMARY,index_users_on_chat_id,index_users_login_facebook_id,index_users_on_login,index_users_on_parent_id,index_users_account_type,index_users_email_digest,index_users_id | NULL | NULL | NULL | 31847 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | trusted_users | ref | index_trusted_users_user,index_trusted_users_trust_user | index_trusted_users_trust_user | 5 | messenger_dev.users.id | 6 | Using where |
| 1 | PRIMARY | facebook_friends | index_merge | index_facebook_friends_user,index_facebook_friends_friend | index_facebook_friends_user,index_facebook_friends_friend | 5,5 | NULL | 2 | Using union(index_facebook_friends_user,index_facebook_friends_friend); Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | address_book_contacts | ref | index_address_book_contacts_owner_id,index_address_book_contacts_email | index_address_book_contacts_email | 767 | messenger_dev.users.email_digest | 1 | Using where |
| 1 | PRIMARY | friends | index_merge | index_friends_me_him,index_friends_me,index_friends_him | index_friends_him,index_friends_me | 5,5 | NULL | 18 | Using union(index_friends_him,index_friends_me); Using where; Using join buffer (Block Nested Loop) |
| 2 | SUBQUERY | friends | index_merge | index_friends_me_him,index_friends_me,index_friends_him | index_friends_him,index_friends_me | 5,5 | NULL | 18 | Using union(index_friends_him,index_friends_me); Using where |
+----+-------------+-----------------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+---------+----------------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0,00 sec)
Edit 2: Table structure
CREATE TABLE `address_book_contacts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`email_digest` varchar(191) DEFAULT NULL,
`code` varchar(191) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`owner_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_address_book_contacts_owner_id` (`owner_id`),
KEY `index_address_book_contacts_email` (`email_digest`)
) ENGINE=InnoDB AUTO_INCREMENT=1598109 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `trusted_users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`friend_id` int(11) DEFAULT NULL,
`user_id` int(11) DEFAULT NULL,
`trust_user_id` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `index_trusted_users_on_friend_id` (`friend_id`),
KEY `index_trusted_users_user` (`user_id`),
KEY `index_trusted_users_trust_user` (`trust_user_id`),
CONSTRAINT `fk_rails_007c31c802` FOREIGN KEY (`trust_user_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_ca24cb4e23` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=275576 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `facebook_friends` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) DEFAULT NULL,
`friend_user_id` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`code` varchar(191) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_facebook_friends_user` (`user_id`),
KEY `index_facebook_friends_friend` (`friend_user_id`),
KEY `index_facebook_friends_code` (`code`(5)),
CONSTRAINT `fk_rails_78285a074e` FOREIGN KEY (`friend_user_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_aa3ac53a81` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1149 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `friends` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`me_id` int(11) DEFAULT NULL,
`him_id` int(11) DEFAULT NULL,
`owner_id` int(11) DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`me_status` int(11) DEFAULT '0',
`him_status` int(11) DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `index_friends_me_him` (`me_id`,`him_id`),
KEY `index_friends_me` (`me_id`),
KEY `index_friends_him` (`him_id`),
KEY `index_friends_owner` (`owner_id`),
CONSTRAINT `fk_rails_9fa3474d31` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_d3ebb6657f` FOREIGN KEY (`him_id`) REFERENCES `users` (`id`),
CONSTRAINT `fk_rails_fccfd1b821` FOREIGN KEY (`me_id`) REFERENCES `users` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=95724 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`account_type` varchar(191) NOT NULL,
`firstname` varchar(191) DEFAULT NULL,
`lastname` varchar(191) DEFAULT NULL,
`login` varchar(191) DEFAULT NULL,
`avatar` varchar(191) DEFAULT NULL,
`gender` varchar(1) DEFAULT NULL,
`locale` varchar(191) DEFAULT NULL,
`birthdate` date DEFAULT NULL,
`password_digest` varchar(191) DEFAULT NULL,
`email_digest` varchar(191) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `index_users_on_login` (`login`),
KEY `index_users_account_type` (`account_type`),
KEY `index_users_email_digest` (`email_digest`),
KEY `index_uses_firstname` (`firstname`),
KEY `index_users_lastname` (`lastname`)
) ENGINE=InnoDB AUTO_INCREMENT=32516 DEFAULT CHARSET=utf8mb4;
It looks like MySQL is not choosing any of the available indicies for the users table.
First, run ANALYZE TABLE users;, then re-run the EXPLAIN command. Is the value of the first rows cell now substantially lower than 31847? If so, your problem should be solved!
If not, run OPTIMIZE TABLE users;, then re-run the EXPLAIN command. Is the value of the first rows cell now substantially lower than 31847? If so, your problem should be solved!
If neither of those steps help, try adding USE INDEX (PRIMARY) or USE INDEX (users_id) immediately after the FROM users portion of your query.
Hope this helps!

effective query on a table with over 1 million record

I am really struggling with the following two queries. I have two tables both with over a million records. The first query runs with 30 seconds and the second one runs with over 7 mins.
Basically I would like to get the count class.id and scan_layers.id based on the Lot. What would be the most effective way to do it?
select count(class.id), count(scan_layers.id), Lot, Verified from class left join scan_layers on (class.ScanLayer = scan_layers.id) group by Lot;
select count(class.id), count(distinct scan_layers.id), Lot, Verified from class left join scan_layers on (class.ScanLayer = scan_layers.id) group by Lot;
As I explain it, they both giving me the same explain.
+----+-------------+--------------------+--------+---------------+------------------------+---------+----------------------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------+--------+---------------+------------------------+---------+----------------------------------+---------+----------------------------------------------+
| 1 | SIMPLE | class | index | NULL | defects_scan_layers_fk | 4 | NULL | 4417159 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | scan_layers | eq_ref | PRIMARY | PRIMARY | 4 | cdb.class.ScanLayer | 1 | |
+----+-------------+--------------------+--------+---------------+------------------------+---------+----------------------------------+---------+----------------------------------------------+
CREATE TABLE `class` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`ScanLayer` int(10) unsigned NOT NULL,
`Type` enum('Regular','Critical') NOT NULL DEFAULT 'Regular',
PRIMARY KEY (`id`),
UNIQUE KEY `Defect_UNIQUE` (`ScanLayer`),
KEY `class_scan_layers_fk` (`ScanLayer`),
CONSTRAINT `class_scan_layers_fk` FOREIGN KEY (`ScanLayer`) REFERENCES `scan_layers` (`id`) ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `scan_layers` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`LayerInfo` int(10) unsigned NOT NULL,
`Lot` varchar(45) NOT NULL DEFAULT 'DEFAULT',
`PhysicalID` int(10) unsigned NOT NULL,
`Scanned` datetime DEFAULT NULL,
`ScannedMachine` int(10) unsigned DEFAULT NULL,
`DefectsCount` int(10) unsigned NOT NULL DEFAULT '0',
`MovesCount` int(10) unsigned NOT NULL DEFAULT '0',
`Verified` datetime DEFAULT NULL,
`VerifiedMachine` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ScanLayer_UNIQUE` (`LayerInfo`,`Lot`,`PhysicalID`),
KEY `scan_layers_layer_infos_fk` (`LayerInfo`),
KEY `scan_layers_scanned_machines_fk` (`ScannedMachine`),
KEY `scan_layers_verified_machines_fk` (`VerifiedMachine`),
KEY `scan_layers_verified` (`Verified`),
KEY `scan_layers_lot` (`Lot`),
CONSTRAINT `scan_layers_layer_infos_fk` FOREIGN KEY (`LayerInfo`) REFERENCES `layer_infos` (`id`) ON UPDATE CASCADE,
CONSTRAINT `scan_layers_scanned_machines_fk` FOREIGN KEY (`ScannedMachine`) REFERENCES `machines` (`id`) ON UPDATE CASCADE,
CONSTRAINT `scan_layers_verified_machines_fk` FOREIGN KEY (`VerifiedMachine`) REFERENCES `machines` (`id`) ON UPDATE CASCADE,
) ENGINE=InnoDB
Thanks a lot!

Why is MySQL query using join buffer?

The following query is using the join buffer and I was wondering if someone could explain to me why this is so. Just trying to gain more understanding about mysql and indexing.
mysql> EXPLAIN SELECT events.event_topic_id, event_topic_name, event_topic_image, event_type_name,city_name FROM events
-> JOIN event_topic ON event_topic.event_topic_id=events.event_topic_id
-> JOIN event_type ON event_type.event_type_id = event_topic.event_type_id
-> JOIN locations ON locations.location_id=events.location_id
-> JOIN city ON city.city_id=locations.city_id
-> WHERE event_date > NOW()
-> GROUP BY events.event_topic_id, city.city_id;
+----+-------------+-------------+--------+---------------------------------------+-----------------+---------+--------------------------------------+------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------------+--------+---------------------------------------+-----------------+---------+--------------------------------------+------+----------+----------------------------------------------+
| 1 | SIMPLE | city | index | PRIMARY | city_name | 52 | NULL | 6 | 100.00 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | locations | ref | PRIMARY,city_id | city_id | 1 | PremiumCONNECT.city.city_id | 1 | 100.00 | Using index |
| 1 | SIMPLE | events | ref | location_id,event_topic_id,event_date | location_id | 2 | PremiumCONNECT.locations.location_id | 3 | 100.00 | Using where |
| 1 | SIMPLE | event_type | index | PRIMARY | event_type_name | 52 | NULL | 2 | 100.00 | Using index; Using join buffer |
| 1 | SIMPLE | event_topic | eq_ref | PRIMARY,event_type_id | PRIMARY | 1 | PremiumCONNECT.events.event_topic_id | 1 | 100.00 | Using where |
+----+-------------+-------------+--------+---------------------------------------+-----------------+---------+--------------------------------------+------+----------+----------------------------------------------+
Events table:
CREATE TABLE `events` (
`event_id` smallint(8) unsigned NOT NULL AUTO_INCREMENT,
`location_id` smallint(3) unsigned NOT NULL,
`event_date` datetime NOT NULL,
`event_topic_id` tinyint(3) unsigned NOT NULL,
PRIMARY KEY (`event_id`),
KEY `location_id` (`location_id`),
KEY `event_topic_id` (`event_topic_id`),
KEY `event_date` (`event_date`),
CONSTRAINT `events_ibfk_2` FOREIGN KEY (`event_topic_id`) REFERENCES `event_topic` (`event_topic_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `events_ibfk_3` FOREIGN KEY (`location_id`) REFERENCES `locations` (`location_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=91 DEFAULT CHARSET=latin1
Event topic table:
CREATE TABLE `event_topic` (
`event_topic_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`event_topic_name` varchar(100) DEFAULT NULL,
`event_topic_description` text NOT NULL,
`event_topic_cost` decimal(7,2) DEFAULT NULL,
`event_type_id` tinyint(3) unsigned NOT NULL,
`event_topic_clickthrough` tinytext,
`event_topic_length` varchar(6) NOT NULL,
`event_topic_image` varchar(41) DEFAULT NULL,
`event_topic_image_md5` char(32) NOT NULL,
PRIMARY KEY (`event_topic_id`),
KEY `event_type_id` (`event_type_id`),
KEY `topic_image_sha1` (`event_topic_image_md5`),
CONSTRAINT `event_topic_ibfk_1` FOREIGN KEY (`event_type_id`) REFERENCES `event_type` (`event_type_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1
Event type table:
CREATE TABLE `event_type` (
`event_type_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`event_type_name` varchar(50) NOT NULL,
`conf_email` text,
PRIMARY KEY (`event_type_id`),
KEY `event_type_name` (`event_type_name`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1
Locations table:
CREATE TABLE `locations` (
`location_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`location_name` varchar(50) NOT NULL,
`location_address` tinytext NOT NULL,
`location_capacity` smallint(6) NOT NULL,
`city_id` tinyint(3) unsigned NOT NULL,
`gps_coords` varchar(30) DEFAULT NULL,
PRIMARY KEY (`location_id`),
KEY `city_id` (`city_id`),
CONSTRAINT `locations_ibfk_1` FOREIGN KEY (`city_id`) REFERENCES `city` (`city_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=latin1
Cities table:
CREATE TABLE `city` (
`city_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`city_name` varchar(50) NOT NULL,
PRIMARY KEY (`city_id`),
UNIQUE KEY `city_name` (`city_name`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1
As it says in 'http://dev.mysql.com/doc/refman/5.1/en/explain-output.html': "Tables from earlier joins are read in portions into the join buffer, and then their rows are used from the buffer to perform the join with the current table."
So in your case, you had already joined event_topic, so the optimizer was able to use event_topic content from the join buffer.
Using a buffer is a good thing; you probably noticed the undesirable "Using temporary; Using filesort" on the first line of EXPLAIN output, which is probably from the GROUP BY and is probably unavoidable in this case.
By the way, will you run into problems with the "UNIQUE" constraint on city_name? I'm thinking of Springfield (two in New Jersey), Washington, Greenville, etc.
Try using:
"STRAIGHT_JOIN" and "FORCE INDEX":
EXPLAIN SELECT events.event_topic_id, event_topic_name, event_topic_image, event_type_name,city_name FROM events
-> straight_join event_topic force index(primary) ON event_topic.event_topic_id=events.event_topic_id
-> straight_join event_type force index(primary) ON event_type.event_type_id = event_topic.event_type_id
-> straight_join locations force index(primary) ON locations.location_id=events.location_id
-> straight_join city force index(primary) ON city.city_id=locations.city_id
-> WHERE event_date > NOW()
-> GROUP BY events.event_topic_id, city.city_id;
BTW, using a join buffer is not good. It means that you need to improve or reference to correct index.

MySQL using index when joining two tables

any help would be appreciated.
I have 2 tables:
CREATE TABLE `users` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`username` varchar(255) NOT NULL,
`username_canonical` varchar(255) NOT NULL,
`email` varchar(255) NOT NULL,
`email_canonical` varchar(255) NOT NULL,
`enabled` tinyint(1) NOT NULL,
`salt` varchar(255) NOT NULL,
`password` varchar(255) NOT NULL,
`last_login` datetime DEFAULT NULL,
`locked` tinyint(1) NOT NULL,
`expired` tinyint(1) NOT NULL,
`expires_at` datetime DEFAULT NULL,
`confirmation_token` varchar(255) DEFAULT NULL,
`password_requested_at` datetime DEFAULT NULL,
`roles` longtext NOT NULL COMMENT '(DC2Type:array)',
`credentials_expired` tinyint(1) NOT NULL,
`credentials_expire_at` datetime DEFAULT NULL,
`messages_unread_count` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_1483A5E992FC23A8` (`username_canonical`),
UNIQUE KEY `UNIQ_1483A5E9A0D96FBF` (`email_canonical`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8
and
CREATE TABLE `blog_posts` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`user_id` bigint(20) DEFAULT NULL,
`category_id` int(11) DEFAULT NULL,
`title` varchar(255) NOT NULL,
`content` longtext NOT NULL,
`tags` varchar(512) NOT NULL,
`timestamp_created` datetime NOT NULL,
`timestamp_updated` datetime NOT NULL,
`is_visible` tinyint(1) NOT NULL,
`is_deleted` tinyint(1) NOT NULL,
PRIMARY KEY (`id`),
KEY `IDX_78B2F93212469DE2` (`category_id`),
KEY `IDX_78B2F932A76ED395` (`user_id`),
CONSTRAINT `FK_78B2F932A76ED395` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`),
CONSTRAINT `FK_78B2F93212469DE2` FOREIGN KEY (`category_id`) REFERENCES `blog_categories` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8
and I want to select all post limiting the output with posts authors, give me an advice - how could I force the query to use indexes:
select p.*, u.* from blog_posts p join users u on p.user_id = u.id limit 0,10;
Explain output:
+----+-------------+-------+------+----------------------+----------------------+---------+----------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+----------------------+----------------------+---------+----------+------+-------------+
| 1 | SIMPLE | u | ALL | PRIMARY | NULL | NULL | NULL | 1 | |
| 1 | SIMPLE | p | ref | IDX_78B2F932A76ED395 | IDX_78B2F932A76ED395 | 9 | sky.u.id | 1 | Using where |
+----+-------------+-------+------+----------------------+----------------------+---------+----------+------+-------------+
Is there any way to use index when joining users table?
Mysql will use an index automatically if you setup a FK relationship between blogposts.userid and and users.id.
So create the FK relationship:
CONSTRAINT blogposts_userid FOREIGN KEY (user_id) REFERENCES users (id)