Avoid filesort with INNER JOIN + ORDER BY - mysql

I've been reading other posts but I didn't managed to fix my query.
Using DESC order the query is x20 times slower, I must improve that.
This is the query:
SELECT posts.post_id, posts.post_b_id, posts.post_title, posts.post_cont, posts.thumb, posts.post_user, boards.board_title_l, boards.board_title
FROM posts
INNER JOIN follow ON posts.post_b_id = follow.board_id
INNER JOIN boards ON posts.post_b_id = boards.board_id
WHERE follow.user_id =1
ORDER BY posts.post_id DESC
LIMIT 10
And these are the tables (Updated):
CREATE TABLE IF NOT EXISTS `posts` (
`post_id` int(11) NOT NULL AUTO_INCREMENT,
`post_b_id` int(11) unsigned NOT NULL,
`post_title` varchar(50) COLLATE utf8_bin NOT NULL,
`post_cont` text COLLATE utf8_bin NOT NULL,
`post_mintxt` varchar(255) COLLATE utf8_bin NOT NULL,
`post_type` char(3) COLLATE utf8_bin NOT NULL,
`thumb` varchar(200) COLLATE utf8_bin NOT NULL,
`post_user` varchar(16) COLLATE utf8_bin NOT NULL,
`published` enum('0','1') COLLATE utf8_bin NOT NULL,
`post_ip` varchar(94) COLLATE utf8_bin NOT NULL,
`post_ip_dat` int(11) unsigned NOT NULL,
`post_up` int(10) unsigned NOT NULL DEFAULT '0',
`post_down` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`post_id`),
KEY `post_b_id` (`post_b_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=405 ;
CREATE TABLE IF NOT EXISTS `boards` (
`board_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`board_title_l` varchar(19) COLLATE utf8_bin NOT NULL,
`board_user_id` int(10) unsigned NOT NULL,
`board_title` varchar(19) COLLATE utf8_bin NOT NULL,
`board_user` varchar(16) COLLATE utf8_bin NOT NULL,
`board_txt` tinyint(1) unsigned NOT NULL,
`board_img` tinyint(1) unsigned NOT NULL,
`board_vid` tinyint(1) unsigned NOT NULL,
`board_desc` varchar(100) COLLATE utf8_bin NOT NULL,
`board_mod_p` tinyint(3) unsigned NOT NULL DEFAULT '0',
`board_ip` varchar(94) COLLATE utf8_bin NOT NULL,
`board_dat_ip` int(11) unsigned NOT NULL,
PRIMARY KEY (`board_id`),
UNIQUE KEY `board_title_l` (`board_title_l`),
KEY `board_user_id` (`board_user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=89 ;
CREATE TABLE IF NOT EXISTS `follow` (
`user_id` int(10) unsigned NOT NULL,
`board_id` int(10) unsigned NOT NULL,
PRIMARY KEY (`user_id`,`board_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Using default ASC order it only uses index and where, with DESC uses index, where, temporary and filesort.
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE follow ref user_id user_id 4 const 2 100.00 Using index; Using temporary; Using filesort
1 SIMPLE boards eq_ref PRIMARY PRIMARY 4 xxxx.follow.board_id 1 100.00
1 SIMPLE posts ref post_b_id post_b_id 4 xxxx.boards.board_id 3 100.00 Using where
How I can make the query receiving the results in DESC order without filesort and temporary.
UPDATE: I made a new query, no temporary or filesort, but type: index, filtered: 7340.00. Almost as fast as ASC order if the posts are at the end of the table, but slow if the posts that is searching are at the beginning. So seems better but it's not enough.
SELECT posts.post_id, posts.post_b_id, posts.post_title, posts.post_cont, posts.thumb, posts.post_user, boards.board_title_l, boards.board_title
FROM posts INNER JOIN boards ON posts.post_b_id = boards.board_id
WHERE posts.post_b_id
IN (
SELECT follow.board_id
FROM follow
WHERE follow.user_id = 1
)
ORDER BY posts.post_id DESC
LIMIT 10
Explain:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY posts index post_b_id PRIMARY 8 NULL 10 7340.00 Using where
1 PRIMARY boards eq_ref PRIMARY PRIMARY 4 xxxx.posts.post_b_id 1 100.00
2 DEPENDENT SUBQUERY follow eq_ref user_id user_id 8 const,func 1 100.00 Using index
UPDATE: Explain for the query from dened's answer:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2>ALL NULL NULL NULL NULL 10 100.00
1 PRIMARY posts eq_ref PRIMARY,post_b_id PRIMARY 4 sq.post_id 1 100.00
1 PRIMARY boards eq_ref PRIMARY PRIMARY 4 xxxx.posts.post_b_id 1 100.00
2 DERIVED follow ref PRIMARY PRIMARY 4 1 100.00 Using index; Using temporary; Using filesort
2 DERIVED posts ref post_b_id post_b_id 4 xxxx.follow.board_id 6 100.00 Using index
Times:
Original query no order (ASC): 0.187500 seconds
Original query DESC: 2.812500 seconds
Second query posts at the end (DESC): 0.218750 seconds
Second query posts at the beginning (DESC): 3.293750 seconds
dened's query DESC: 0.421875 seconds
dened's query no order (ASC): 0.323750 seconds
Interesting note, if I add ORDER BY ASC is as slow as DESC.
Alter the table order will be a god way, but as I said in the comments I wasn't able to do that.

You can help MySQL optimizer by moving all the filtering work to a subquery that accesses only indices (manipulating indices is usually much faster than manipulating other data), and fetching rest of the data in the outermost query:
SELECT posts.post_id,
posts.post_b_id,
posts.post_title,
posts.post_cont,
posts.thumb,
posts.post_user,
boards.board_title_l,
boards.board_title
FROM (SELECT post_id
FROM posts
JOIN follow
ON posts.post_b_id = follow.board_id
WHERE follow.user_id = 1
ORDER BY post_id DESC
LIMIT 10) sq
JOIN posts
ON posts.post_id = sq.post_id
JOIN boards
ON boards.board_id = posts.post_b_id
Note that I omit ORDER BY posts.post_id DESC from the outer query, because it is usually faster to sort the final result in your code rather than sorting using a MySQL query (MySQL often uses filesort for that).
P.S. You can replace the unique key in the follow table with a primary key.

Increasing the sort_buffer_size parameter will increase the amount of memory MySQL uses before resorting to a temporary disk file and should help considerably.

Related

Why is subquery join much faster than direct join

I have 2 tables (pages and comments), around 130 000 rows each.
I want to list pages without any comments (foreign key is comments.page_id)
If I execute the normal left outer join, it takes an amazing more than 750 seconds to run. (130k^2 = 17B). Whereas if I execute the same join, but using subqueries for the tables, it takes just 1 second.
Server version: 5.6.44-log - MySQL Community Server (GPL):
Query 1. Normal join, 750+ seconds
SELECT p.id
FROM `pages` AS p
LEFT JOIN `comments` AS c
ON p.id = c.page_id
WHERE c.page_id IS NULL
GROUP BY 1
Query 2. Join with first table as subquery, Too much time
SELECT p.id
FROM (
SELECT id FROM `pages`
) AS p
LEFT JOIN `comments` AS c
ON p.id = c.page_id
WHERE c.page_id IS NULL
GROUP BY 1
Query 3. Join with second table as subquery, 1.6 seconds
SELECT p.id
FROM `pages` AS p
LEFT JOIN (
SELECT * FROM `comments`
) AS c
ON p.id = c.page_id
WHERE c.page_id IS NULL
GROUP BY 1
Query 4. Join with 2 subqueries, 1 second
SELECT p.id
FROM (
SELECT id FROM `pages`
) AS p
LEFT JOIN (
SELECT * FROM `comments`
) AS c
ON p.id = c.page_id
WHERE c.page_id IS NULL
GROUP BY 1
Query 5. Join with 2 subqueries, selecting only 1 column, 0.2 seconds
SELECT p.id
FROM (
SELECT id FROM `pages`
) AS p
LEFT JOIN (
SELECT page_id FROM `comments`
) AS c
ON p.id = c.page_id
WHERE c.page_id IS NULL
GROUP BY 1
Query 6. Too much time
SELECT p.id
FROM `pages` AS p
WHERE NOT EXISTS( SELECT page_id FROM `comments`
WHERE page_id = p.id );;
Now, in MySql version 5.7, all of the above queries take "too much time" to execute.
In MySql 5.7, query 1 and 4 have same explanation:
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 SIMPLE p NULL index PRIMARY PRIMARY 4 NULL 147626 100.00 Using index; Using temporary; Using filesort
1 SIMPLE c NULL ALL NULL NULL NULL NULL 147790 10.00 Using where; Not exists; Using join buffer (Block Nested Loop)
In MySql 5.6, unfortunately I cannot get the explanation for query 1 right now (taking too much time), but for query 4 is the below:
id select_type table type possible_keys key key_len ref rows Extra
---------------------------------------------------------------------------------------------------------------------------
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 147626 Using temporary; Using filesort
1 PRIMARY <derived3> ref <auto_key0> <auto_key0> 4 p.id 10 Using where; Not exists
3 DERIVED comments ALL NULL NULL NULL NULL 147790 NULL
2 DERIVED pages index NULL PRIMARY 4 NULL 147626 Using index
Tables:
CREATE TABLE `pages` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`identifier` varchar(250) NOT NULL DEFAULT '',
`reference` varchar(250) NOT NULL DEFAULT '',
`url` varchar(1000) NOT NULL DEFAULT '',
`moderate` varchar(250) NOT NULL DEFAULT 'default',
`is_form_enabled` tinyint(1) unsigned NOT NULL DEFAULT '1',
`date_modified` datetime NOT NULL,
`date_added` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=147627 DEFAULT CHARSET=utf8
CREATE TABLE `comments` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(10) unsigned NOT NULL DEFAULT '0',
`page_id` int(10) unsigned NOT NULL DEFAULT '0',
`website` varchar(250) NOT NULL DEFAULT '',
`town` varchar(250) NOT NULL DEFAULT '',
`state_id` int(10) NOT NULL DEFAULT '0',
`country_id` int(10) NOT NULL DEFAULT '0',
`rating` tinyint(1) unsigned NOT NULL DEFAULT '0',
`reply_to` int(10) unsigned NOT NULL DEFAULT '0',
`comment` text NOT NULL,
`reply` text NOT NULL,
`ip_address` varchar(250) NOT NULL DEFAULT '',
`is_approved` tinyint(1) unsigned NOT NULL DEFAULT '1',
`notes` text NOT NULL,
`is_admin` tinyint(1) unsigned NOT NULL DEFAULT '0',
`is_sent` tinyint(1) unsigned NOT NULL DEFAULT '0',
`sent_to` int(10) unsigned NOT NULL DEFAULT '0',
`likes` int(10) unsigned NOT NULL DEFAULT '0',
`dislikes` int(10) unsigned NOT NULL DEFAULT '0',
`reports` int(10) unsigned NOT NULL DEFAULT '0',
`is_sticky` tinyint(1) unsigned NOT NULL DEFAULT '0',
`is_locked` tinyint(1) unsigned NOT NULL DEFAULT '0',
`is_verified` tinyint(1) unsigned NOT NULL DEFAULT '0',
`date_modified` datetime NOT NULL,
`date_added` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=147879 DEFAULT CHARSET=utf8
Questions
Why is this happening? What does MySql do under the hood?
Does this happen only in MySql, or any other Sql as well?
How can I write a fast query to get what I need? (In both v 5.6, 5.7)
The problem with your long-running queries, is that you lack an index on the page_id column of the comments table. Hence, for each row from the pages table, you need to check all rows of the comments table. Since you are using LEFT JOIN, this is the only possible join order. What happens in 5.6, is that when you use a subquery in the FROM clause (aka derived table), MySQL will create an index on the temporary table used for the result of the derived table (auto_key0 in the EXPLAIN output). The reason it is faster when you only select one column, is that the temporary table will be smaller.
In MySQL 5.7, such derived tables will be automatically merge into the main query, if possible. This is done to avoid the extra temporary tables. However, this means that you no longer have an index to use for the join. (See this blog post for details.)
You have two options to improve the query time in 5.7:
You can create an index on comments(page_id)
You can prevent the subquery from being merged by rewriting it to a query that can not be merged. Subqueries with aggregation, LIMIT, or UNION will not be merged (see the blog post for details). One way to do this is to add a LIMIT clause to the subquery. In order not to remove any rows from the result, the limit must be larger than the number of rows in the table.
In MySQL 8.0, you can also use an optimizer hint to avoid the merging. In your case, that would be something like
SELECT /*+ NO_MERGE(c) */ ... FROM
See slides 34-37 of this presentation for examples of how to use such hints.
Query 1 has the "explode-implode" syndrome. First it does a JOIN; this explodes the number of rows. Then it does a GROUP BY to shrink back.
Also
The number of comments per page, etc, will have an effect on your query.
SELECT * fetches all the columns, when it needs only to know if the LEFT JOIN succeeded. (You observed that.) Furthermore, you keep none of the columns since you are looking for missing rows.
Query 2 should not be as fast as what you found -- It needs to build two temp tables (the "derived" tables), index one of them, then perform the outer query. (Possibly a new enough version of MySQL can short-circuit some of that effort; old version were notorious at doing an inefficient job.)
Query 3:
Try
SELECT p.id
FROM `pages` AS p
WHERE NOT EXISTS( SELECT 1 FROM `comments`
WHERE page_id = p.id );
ALSO:
Use InnoDB, not MyISAM.
comments needs INDEX(page_id)

MySQL Query Using LEFT JOIN Hangs

Can somebody please advise where I could be going wrong with the following query which is causing my page to hang:
SELECT m.member_id, m.member_name, SUM(CONCAT(f.pounds, '.', f.pence)) AS totalfigure
FROM members m
LEFT JOIN figures f ON f.member_id = m.member_id AND f.period_id = 45
WHERE m.added BETWEEN '1980-01-01' AND '2014-10-31'
AND (DATE_SUB('2014-08-01', INTERVAL 60 DAY)) < m.removed
GROUP BY m.member_id
ORDER BY m.member_name;
The same query runs fine if I use JOIN rather than LEFT JOIN but I need to also view members that do not have an entry in the figures table.
This is the EXPLAIN result I get with the LEFT JOIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE m ALL NULL NULL NULL NULL 3128 Using temporary; Using filesort
1 SIMPLE f ALL NULL NULL NULL NULL 169214
and with the JOIN the result is:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE f ALL NULL NULL NULL NULL 169214 Using where; Using temporary; Using filesort
1 SIMPLE m eq_ref PRIMARY PRIMARY 3 db.f.member_id 1
The structure for each of these tables is:
CREATE TABLE IF NOT EXISTS `members` (
`member_id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`member_name` varchar(100) NOT NULL DEFAULT '',
PRIMARY KEY (`member_id`),
FULLTEXT KEY `member_name` (`member_name`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=3462 ;
CREATE TABLE IF NOT EXISTS `figures` (
`supplier_id` int(10) unsigned NOT NULL DEFAULT '0',
`contract_id` mediumint(5) unsigned NOT NULL DEFAULT '1',
`member_id` mediumint(5) unsigned NOT NULL,
`period_id` mediumint(5) unsigned NOT NULL DEFAULT '0',
`pounds` varchar(8) DEFAULT NULL,
`pence` char(3) DEFAULT '00',
PRIMARY KEY (`supplier_id`,`member_id`,`period_id`,`contract_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Thanks

MYSQL, very slow order by

I have got two tables. One is a User table with a primary key on the userid and the other table references the user table with a foreign key.
The User table has only one entry (for now) and the other table has one million entrys.
The following join drives me mad:
SELECT p0_.*, p1_.*
FROM photo p0_, User p1_
WHERE p0_.user_id = p1_.user_id
ORDER BY p0_.uploaddate DESC Limit 10 OFFSET 100000
The query takes 12sec on a very fast machine with the order by and 0.0005 sec without the order by.
I've got an index on user_id (IDX_14B78418A76ED395) and a composite index ("search2") on user_id and uploaddate.
EXPLAIN shows the following:
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
| 1 | SIMPLE | p1_ | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | p0_ | ref | IDX_14B78418A76ED395,search2 | IDX_14B78418A76ED395 | 4 | odsfoto.p1_.user_id | 58520 | |
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
Table definitions:
CREATE TABLE `photo` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`album_id` int(11) DEFAULT NULL,
`exif_id` int(11) DEFAULT NULL,
`title` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`width` int(11) NOT NULL,
`height` int(11) NOT NULL,
`uploaddate` datetime NOT NULL,
`filesize` int(11) DEFAULT NULL,
`path` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`originalFilename` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`mimeType` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`description` longtext COLLATE utf8_unicode_ci,
`gpsData_id` int(11) DEFAULT NULL,
`views` int(11) DEFAULT NULL,
`likes` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_14B78418B0FC9251` (`exif_id`),
UNIQUE KEY `UNIQ_14B7841867E96507` (`gpsData_id`),
KEY `IDX_14B78418A76ED395` (`user_id`),
KEY `IDX_14B784181137ABCF` (`album_id`),
KEY `search_idx` (`uploaddate`),
KEY `search2` (`user_id`,`uploaddate`),
KEY `search3` (`uploaddate`,`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `user` (
`user_id` int(11) NOT NULL,
`photoCount` int(11) NOT NULL,
`photoViews` int(11) NOT NULL,
`photoComments` int(11) NOT NULL,
`photoLikes` int(11) NOT NULL,
`username` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
What can I do to speed up this query?
Seems you're suffering from MySQL's inability to do late row lookups:
MySQL ORDER BY / LIMIT performance: late row lookups
Late row lookups: InnoDB
Try this:
SELECT p.*, u.*
FROM (
SELECT id
FROM photo
ORDER BY
uploaddate DESC, id DESC
LIMIT 10
OFFSET 100000
) pi
JOIN photo p
ON p.id = pi.id
JOIN user u
ON u.user_id = p.user_id
You need a separate index on uploaddate. This sort will take advantage of composite index only if uploaddate is first column in it.
You can also try to add user_id to ORDER BY:
....
ORDER BY p0_.user_id, p0_.uploaddate
You have two problems:
You need to create an INDEX(user_id, uploaddate) which will greatly increase the efficiency of the query.
You need to find a workaround to using LIMIT 10 OFFSET 100000. MySQL is creating a recordset with 100,000 records in it, then it pulls the last 10 records off the end... that is extremely inefficient.
https://www.percona.com/blog/2006/09/01/mysql-order-by-limit-performance-optimization/
First try to get result based on primary key with out join and use result to query result again.
For ex:
$userIds=mysql::select("select user_id from photo ORDER BY p0_.uploaddate DESC Limit 10 OFFSET 100000");
$photoData=mysql::select("SELECT p0_., p1_.
FROM photo p0_, User p1_
WHERE p0_.user_id = p1_.user_id and p0_.user_id in ($userIds->user_id) order by p0_.uploaddate");
Here we had divided the statement into two parts:
1.We can easily order and get based on primary key and also there are no joins.
2.Getting query results based on id and order by is only on limited columns we can retrieve data in less time
From 30 seconds to 0.015 sec / 0.000 sec using Quassnoi answer !
This is what I called MySql expertise !
I cut out one Join from my personal project (no join with itself)
Select ser.id_table, ser.id_rec, ser.relevance, cnt, title, description, sell_url, medium_thumb,
unique_id_supplier, keywords width, height, media_type
from (
Select ser.id_rec, ser.id_table, ser.relevance, ser.cnt
from searchEngineResults ser
where thisSearch = 16287
order by ser.relevance desc, cnt desc, id_rec
) ser
join photo_resell sou on sou.id = ser.id_rec
#join searchEngineResults ser on ser.id_rec = tmp.id_rec
limit 0, 9

MySQL query optimization - increase speed and efficiency

I want to get some emails in a database and each email have a status. All the possible status are stock in a table where they all have permissions (such as show, edit, delete, etc.). Those emails are not users with permissions trough a site but a list of emails a user have added.
Here is the tables structure:
Email table
CREATE TABLE IF NOT EXISTS `email__email` (
`email_id` int(11) NOT NULL AUTO_INCREMENT,
`created` timestamp NULL DEFAULT NULL,
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`user_fk` int(11) NOT NULL,
`status_fk` tinyint(2) NOT NULL,
`language` enum('fr','en') COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`firstName` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastName` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`companyName` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`gender` enum('f','m') COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`email_id`),
UNIQUE KEY `user_email` (`user_fk`,`email`),
KEY `user_fk` (`user_fk`),
KEY `created` (`created`),
KEY `status_fk` (`status_fk`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=3031492 ;
Status table
CREATE TABLE IF NOT EXISTS `email__status` (
`status_id` int(11) NOT NULL AUTO_INCREMENT,
`name_fr` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`name_en` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`description_fr` varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
`description_en` varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
`permShow` tinyint(1) NOT NULL DEFAULT '0',
`permSend` tinyint(1) NOT NULL DEFAULT '0',
`permEdit` tinyint(1) NOT NULL DEFAULT '0',
`permDelete` tinyint(1) NOT NULL DEFAULT '0',
`permImport` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`status_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=7 ;
Here is the slow query with the EXPLAIN:
SELECT EE.*, ES.name_fr AS statusName, ES.description_fr AS statusDescription, ES.permShow, ES.permSend, ES.permEdit, ES.permDelete, ES.permImport
, (SELECT GROUP_CONCAT(CONVERT(CONCAT(GC.name, '~', GC.group_id), CHAR(255)) SEPARATOR ',') FROM `group` GC INNER JOIN group_email GEC ON GEC.group_fk = GC.group_id WHERE GEC.email_fk = EE.email_id AND GC.deleted = 0) AS groups
FROM `email__email` EE
INNER JOIN email__status ES ON EE.status_fk = ES.status_id
WHERE 1 = 1
AND EE.user_fk = 54
AND ES.permShow = 1
ORDER BY EE.email_id DESC LIMIT 15
EXTRA ID KEY KEY_LEN POSSIBLE_KEYS REF ROWS SELECT_TYPE TABLE TYPE
Using temporary; Using filesort 1 user_email 4 user_email,user_fk,status_fk const 180681 PRIMARY EE ref
Using where; Using join buffer 1 [empty string] [empty string] PRIMARY [empty string] 6 PRIMARY ES ALL
Using index 2 email_fk 4 group_email,group_fk,email_fk mailing_dev.EE.email_id 1 DEPENDENT SUBQUERY GEC ref
Using where 2 PRIMARY 4 PRIMARY mailing_dev.GEC.group_fk 1 DEPENDENT SUBQUERY GC eq_ref
Here is a fast query with the EXPLAIN:
SELECT EE.*
, (SELECT GROUP_CONCAT(CONVERT(CONCAT(GC.name, '~', GC.group_id), CHAR(255)) SEPARATOR ',') FROM `group` GC INNER JOIN group_email GEC ON GEC.group_fk = GC.group_id WHERE GEC.email_fk = EE.email_id AND GC.deleted = 0) AS groups
FROM `email__email` EE
WHERE 1 = 1
AND EE.user_fk = 54
AND EXISTS(SELECT permShow FROM email__status WHERE status_id = EE.status_fk AND permShow = 1)
ORDER BY EE.email_id DESC LIMIT 15
EXTRA ID KEY KEY_LEN POSSIBLE_KEYS REF ROWS SELECT_TYPE TABLE TYPE
Using where 1 PRIMARY 4 user_email,user_fk [empty string] 270 PRIMARY EE index
Using where 3 PRIMARY 4 PRIMARY mailing_dev.EE.status_fk 1 DEPENDENT SUBQUERY email__status eq_ref
Using index 2 email_fk 4 group_email,group_fk,email_fk mailing_dev.EE.email_id 1 DEPENDENT SUBQUERY GEC ref
Using where 2 PRIMARY 4 PRIMARY mailing_dev.GEC.group_fk 1 DEPENDENT SUBQUERY GC eq_ref
There is a big difference between both queries but the second one doesn't give me two important columns that I need to fetch. I can do subqueries to fetch them like a join would do but still, I don't want to have a lot of subqueries for each... any ideas to improve this ?
Thanks
email__email.status_fk is a tinyint, but email__status.status_id is an int(11).
This probably is fouling up your INNER JOIN. Change one or the other data type and try again.

MySQL stuck on "using filesort" when doing an "order by"

I can't seem to get my query to stop using filesort.
This is my query:
SELECT s.`pilot`, p.`name`, s.`sector`, s.`hull`
FROM `pilots` p
LEFT JOIN `ships` s ON ( (s.`game` = p.`game`)
AND (s.`pilot` = p.`id`) )
WHERE p.`game` = 1
AND p.`id` <> 2
AND s.`sector` = 43
AND s.`hull` > 0
ORDER BY p.`last_move` DESC
Table structures:
CREATE TABLE IF NOT EXISTS `pilots` (
`id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`game` tinyint(3) unsigned NOT NULL DEFAULT '0',
`last_move` int(10) NOT NULL DEFAULT '0',
UNIQUE KEY `id` (`id`),
KEY `last_move` (`last_move`),
KEY `game_id_lastmove` (`game`,`id`,`last_move`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8 ;
CREATE TABLE IF NOT EXISTS `ships` (
`id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`game` tinyint(3) unsigned NOT NULL DEFAULT '0',
`pilot` mediumint(5) unsigned NOT NULL DEFAULT '0',
`sector` smallint(5) unsigned NOT NULL DEFAULT '0',
`hull` smallint(4) unsigned NOT NULL DEFAULT '50',
UNIQUE KEY `id` (`id`),
KEY `game` (`game`),
KEY `pilot` (`pilot`),
KEY `sector` (`sector`),
KEY `hull` (`hull`),
KEY `game_2` (`game`,`pilot`,`sector`,`hull`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8 ;
The explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE p ref id,game_id_lastmove game_id_lastmove 1 const 7 Using where; Using filesort
1 SIMPLE s ref game,pilot,sector... game_2 6 const,fightclub_alpha.p.id,const 1 Using where; Using index
edit: I cut some of the unnecessary pieces out of my queries/table structure.
Anybody have any ideas?
the best thing that you can do is to make indexes:
index that covers table ships with fields: game + pilot + sector + hull (in this specific order)
pilots: game + id
this particular query will always use filesort, because it has not range condition p.id <> 2
http://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html
In some cases, MySQL cannot use
indexes to resolve the ORDER BY,
although it still uses indexes to find
the rows that match the WHERE clause.
These cases include the following ...
The key used to fetch the rows is not
the same as the one used in the ORDER
BY