MySQL migration from 5.5 to 5.7 extremely slow - mysql

As mentioned in the title, I'm migrating from 5.5 to 5.7, and I have 1 query that totally broke:
SELECT
COUNT(*) as count,
detections.vendor
FROM (
SELECT id FROM telemetry_user_agents
WHERE date < DATE_SUB(NOW(), INTERVAL 0 DAY)
AND date > DATE_SUB(NOW(), INTERVAL 7 DAY)
) agents
LEFT JOIN telemetry_detections detections on detections.user_id = agents.id
GROUP BY detections.vendor
ORDER BY count DESC
LIMIT 20
On 5.5 the query takes 4 seconds, on 5.7 it takes 400 seconds (!), with exact same indexes and data. I've read about bugs in 5.7, but I'm pretty sure I can find a workaround with some query optimization and/or better indexes.
user_agents has 200k records, detections has 900k records
Here's the structures:
CREATE TABLE `telemetry_user_agents` (
`id` int(11) NOT NULL,
`date` datetime NOT NULL,
`program` text NOT NULL,
`country` text NOT NULL,
`ip` text NOT NULL,
`operating_system` text NOT NULL,
`bits` int(11) NOT NULL,
`version` text NOT NULL,
`license` text NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
ALTER TABLE `telemetry_user_agents`
ADD PRIMARY KEY (`id`),
ADD KEY `date` (`date`) USING BTREE;
ALTER TABLE `telemetry_user_agents`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
CREATE TABLE `telemetry_detections` (
`id` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`type` text NOT NULL,
`vendor` text NOT NULL,
`name` text NOT NULL,
`value` text NOT NULL,
`hash` text NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
ALTER TABLE `telemetry_detections`
ADD PRIMARY KEY (`id`),
ADD KEY `user_id` (`user_id`);
ALTER TABLE `telemetry_detections`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
And the explains:
5.5
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 95286 Using temporary; Using filesort
1 PRIMARY detections ref user_id user_id 4 agents.id 19
2 DERIVED telemetry_user_agents ALL date NULL NULL NULL 202047 Using where
5.7
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE telemetry_user_agents NULL ALL date NULL NULL NULL 200866 46.50 Using where; Using temporary; Using filesort
1 SIMPLE detections NULL ref user_id user_id 4 adlice_stats.telemetry_user_agents.id 19 100.00 NULL
Any help? :)
EDIT: After I created a bunch of indexes:
5.7:
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE telemetry_user_agents NULL range date,date_2 date_2 5 NULL 131741 100.00 Using where; Using index; Using temporary; Using f...
1 SIMPLE detections NULL ref user_id user_id 4 adlice_stats.telemetry_user_agents.id 19 100.00 NULL

Related

Avoid filesort with INNER JOIN + ORDER BY

I've been reading other posts but I didn't managed to fix my query.
Using DESC order the query is x20 times slower, I must improve that.
This is the query:
SELECT posts.post_id, posts.post_b_id, posts.post_title, posts.post_cont, posts.thumb, posts.post_user, boards.board_title_l, boards.board_title
FROM posts
INNER JOIN follow ON posts.post_b_id = follow.board_id
INNER JOIN boards ON posts.post_b_id = boards.board_id
WHERE follow.user_id =1
ORDER BY posts.post_id DESC
LIMIT 10
And these are the tables (Updated):
CREATE TABLE IF NOT EXISTS `posts` (
`post_id` int(11) NOT NULL AUTO_INCREMENT,
`post_b_id` int(11) unsigned NOT NULL,
`post_title` varchar(50) COLLATE utf8_bin NOT NULL,
`post_cont` text COLLATE utf8_bin NOT NULL,
`post_mintxt` varchar(255) COLLATE utf8_bin NOT NULL,
`post_type` char(3) COLLATE utf8_bin NOT NULL,
`thumb` varchar(200) COLLATE utf8_bin NOT NULL,
`post_user` varchar(16) COLLATE utf8_bin NOT NULL,
`published` enum('0','1') COLLATE utf8_bin NOT NULL,
`post_ip` varchar(94) COLLATE utf8_bin NOT NULL,
`post_ip_dat` int(11) unsigned NOT NULL,
`post_up` int(10) unsigned NOT NULL DEFAULT '0',
`post_down` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`post_id`),
KEY `post_b_id` (`post_b_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=405 ;
CREATE TABLE IF NOT EXISTS `boards` (
`board_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`board_title_l` varchar(19) COLLATE utf8_bin NOT NULL,
`board_user_id` int(10) unsigned NOT NULL,
`board_title` varchar(19) COLLATE utf8_bin NOT NULL,
`board_user` varchar(16) COLLATE utf8_bin NOT NULL,
`board_txt` tinyint(1) unsigned NOT NULL,
`board_img` tinyint(1) unsigned NOT NULL,
`board_vid` tinyint(1) unsigned NOT NULL,
`board_desc` varchar(100) COLLATE utf8_bin NOT NULL,
`board_mod_p` tinyint(3) unsigned NOT NULL DEFAULT '0',
`board_ip` varchar(94) COLLATE utf8_bin NOT NULL,
`board_dat_ip` int(11) unsigned NOT NULL,
PRIMARY KEY (`board_id`),
UNIQUE KEY `board_title_l` (`board_title_l`),
KEY `board_user_id` (`board_user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=89 ;
CREATE TABLE IF NOT EXISTS `follow` (
`user_id` int(10) unsigned NOT NULL,
`board_id` int(10) unsigned NOT NULL,
PRIMARY KEY (`user_id`,`board_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Using default ASC order it only uses index and where, with DESC uses index, where, temporary and filesort.
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE follow ref user_id user_id 4 const 2 100.00 Using index; Using temporary; Using filesort
1 SIMPLE boards eq_ref PRIMARY PRIMARY 4 xxxx.follow.board_id 1 100.00
1 SIMPLE posts ref post_b_id post_b_id 4 xxxx.boards.board_id 3 100.00 Using where
How I can make the query receiving the results in DESC order without filesort and temporary.
UPDATE: I made a new query, no temporary or filesort, but type: index, filtered: 7340.00. Almost as fast as ASC order if the posts are at the end of the table, but slow if the posts that is searching are at the beginning. So seems better but it's not enough.
SELECT posts.post_id, posts.post_b_id, posts.post_title, posts.post_cont, posts.thumb, posts.post_user, boards.board_title_l, boards.board_title
FROM posts INNER JOIN boards ON posts.post_b_id = boards.board_id
WHERE posts.post_b_id
IN (
SELECT follow.board_id
FROM follow
WHERE follow.user_id = 1
)
ORDER BY posts.post_id DESC
LIMIT 10
Explain:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY posts index post_b_id PRIMARY 8 NULL 10 7340.00 Using where
1 PRIMARY boards eq_ref PRIMARY PRIMARY 4 xxxx.posts.post_b_id 1 100.00
2 DEPENDENT SUBQUERY follow eq_ref user_id user_id 8 const,func 1 100.00 Using index
UPDATE: Explain for the query from dened's answer:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2>ALL NULL NULL NULL NULL 10 100.00
1 PRIMARY posts eq_ref PRIMARY,post_b_id PRIMARY 4 sq.post_id 1 100.00
1 PRIMARY boards eq_ref PRIMARY PRIMARY 4 xxxx.posts.post_b_id 1 100.00
2 DERIVED follow ref PRIMARY PRIMARY 4 1 100.00 Using index; Using temporary; Using filesort
2 DERIVED posts ref post_b_id post_b_id 4 xxxx.follow.board_id 6 100.00 Using index
Times:
Original query no order (ASC): 0.187500 seconds
Original query DESC: 2.812500 seconds
Second query posts at the end (DESC): 0.218750 seconds
Second query posts at the beginning (DESC): 3.293750 seconds
dened's query DESC: 0.421875 seconds
dened's query no order (ASC): 0.323750 seconds
Interesting note, if I add ORDER BY ASC is as slow as DESC.
Alter the table order will be a god way, but as I said in the comments I wasn't able to do that.
You can help MySQL optimizer by moving all the filtering work to a subquery that accesses only indices (manipulating indices is usually much faster than manipulating other data), and fetching rest of the data in the outermost query:
SELECT posts.post_id,
posts.post_b_id,
posts.post_title,
posts.post_cont,
posts.thumb,
posts.post_user,
boards.board_title_l,
boards.board_title
FROM (SELECT post_id
FROM posts
JOIN follow
ON posts.post_b_id = follow.board_id
WHERE follow.user_id = 1
ORDER BY post_id DESC
LIMIT 10) sq
JOIN posts
ON posts.post_id = sq.post_id
JOIN boards
ON boards.board_id = posts.post_b_id
Note that I omit ORDER BY posts.post_id DESC from the outer query, because it is usually faster to sort the final result in your code rather than sorting using a MySQL query (MySQL often uses filesort for that).
P.S. You can replace the unique key in the follow table with a primary key.
Increasing the sort_buffer_size parameter will increase the amount of memory MySQL uses before resorting to a temporary disk file and should help considerably.

MySQL Query Using LEFT JOIN Hangs

Can somebody please advise where I could be going wrong with the following query which is causing my page to hang:
SELECT m.member_id, m.member_name, SUM(CONCAT(f.pounds, '.', f.pence)) AS totalfigure
FROM members m
LEFT JOIN figures f ON f.member_id = m.member_id AND f.period_id = 45
WHERE m.added BETWEEN '1980-01-01' AND '2014-10-31'
AND (DATE_SUB('2014-08-01', INTERVAL 60 DAY)) < m.removed
GROUP BY m.member_id
ORDER BY m.member_name;
The same query runs fine if I use JOIN rather than LEFT JOIN but I need to also view members that do not have an entry in the figures table.
This is the EXPLAIN result I get with the LEFT JOIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE m ALL NULL NULL NULL NULL 3128 Using temporary; Using filesort
1 SIMPLE f ALL NULL NULL NULL NULL 169214
and with the JOIN the result is:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE f ALL NULL NULL NULL NULL 169214 Using where; Using temporary; Using filesort
1 SIMPLE m eq_ref PRIMARY PRIMARY 3 db.f.member_id 1
The structure for each of these tables is:
CREATE TABLE IF NOT EXISTS `members` (
`member_id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`member_name` varchar(100) NOT NULL DEFAULT '',
PRIMARY KEY (`member_id`),
FULLTEXT KEY `member_name` (`member_name`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=3462 ;
CREATE TABLE IF NOT EXISTS `figures` (
`supplier_id` int(10) unsigned NOT NULL DEFAULT '0',
`contract_id` mediumint(5) unsigned NOT NULL DEFAULT '1',
`member_id` mediumint(5) unsigned NOT NULL,
`period_id` mediumint(5) unsigned NOT NULL DEFAULT '0',
`pounds` varchar(8) DEFAULT NULL,
`pence` char(3) DEFAULT '00',
PRIMARY KEY (`supplier_id`,`member_id`,`period_id`,`contract_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Thanks

Optimizing MySQL ORDER BY field of JOINed table

I have the following structure:
CREATE TABLE `tbl1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`tid` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`time` datetime NOT NULL,
`count` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `tid_uid` (`tid`,`user_id`),
KEY `tid_time` (`tid`,`time`)
)
CREATE TABLE `tbl2` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(11) unsigned NOT NULL,
`field_to_order_by` int(11) unsigned NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `user_id` (`user_id`),
KEY `field_to_order_by` (`field_to_order_by`)
)
I'm trying to perform the following query:
SELECT * FROM tbl1
LEFT JOIN tbl2 ON tbl1.user_id=tbl2.user_id
WHERE tbl1.tid=13 AND time > DATE_SUB(NOW(), INTERVAL 1 WEEK)
ORDER BY tbl2.field_to_order_by DESC LIMIT 200
The performance problem I'm facing is due to the ORDER BY of the joined table field. If I remove that or even replace it with a WHERE condition on the same field I'm getting a massive improvement.
How/Can I achieve reasonable performance with this combination of JOIN and ORDER BY or can this only be solved with de-normalization?
This is the EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE tbl1 range tid_uid,tid_time tid_time 12 NULL 221664 Using where; Using temporary; Using filesort
1 SIMPLE tbl2 eq_ref user_id user_id 4 tbl1.user_id 1

Why is MySQL using filesort in this case?

Table Structure:
CREATE TABLE IF NOT EXISTS `newsletters`
(
`id` int(11) NOT NULL auto_increment,
`last_update` int(11) default NULL,
`status` int(11) default '0',
`message_id` varchar(255) default NULL,
PRIMARY KEY (`id`),
KEY `status` (`status`),
KEY `message_id` (`message_id`),
KEY `last_update` (`last_update`)
)
ENGINE=MyISAM DEFAULT CHARSET=latin1;
The Query:
SELECT id, last_update
FROM newsletters
WHERE status = 1
ORDER BY last_update DESC
LIMIT 0, 100
newsletters table has over 3 million records
query takes over 26 seconds to execute
Query explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE newsletters range status status 5 NULL 3043354 Using where; Using filesort
So why is it not using filesort, and how is it a range query?
It's using filesort to sort on last_update. You can avoid the filesort that by changing the index to status, last_update, so MySQL finds all rows with status 1 in the right order.
To further optimize, change the index to status, last_update, id. That allows MySQL to satisfy the query just by looking at the index, without a table lookup.
CREATE INDEX idx_newsletters_status
ON newsletters(status, last_update, id);

MySQL stuck on "using filesort" when doing an "order by"

I can't seem to get my query to stop using filesort.
This is my query:
SELECT s.`pilot`, p.`name`, s.`sector`, s.`hull`
FROM `pilots` p
LEFT JOIN `ships` s ON ( (s.`game` = p.`game`)
AND (s.`pilot` = p.`id`) )
WHERE p.`game` = 1
AND p.`id` <> 2
AND s.`sector` = 43
AND s.`hull` > 0
ORDER BY p.`last_move` DESC
Table structures:
CREATE TABLE IF NOT EXISTS `pilots` (
`id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`game` tinyint(3) unsigned NOT NULL DEFAULT '0',
`last_move` int(10) NOT NULL DEFAULT '0',
UNIQUE KEY `id` (`id`),
KEY `last_move` (`last_move`),
KEY `game_id_lastmove` (`game`,`id`,`last_move`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8 ;
CREATE TABLE IF NOT EXISTS `ships` (
`id` mediumint(5) unsigned NOT NULL AUTO_INCREMENT,
`game` tinyint(3) unsigned NOT NULL DEFAULT '0',
`pilot` mediumint(5) unsigned NOT NULL DEFAULT '0',
`sector` smallint(5) unsigned NOT NULL DEFAULT '0',
`hull` smallint(4) unsigned NOT NULL DEFAULT '50',
UNIQUE KEY `id` (`id`),
KEY `game` (`game`),
KEY `pilot` (`pilot`),
KEY `sector` (`sector`),
KEY `hull` (`hull`),
KEY `game_2` (`game`,`pilot`,`sector`,`hull`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8 ;
The explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE p ref id,game_id_lastmove game_id_lastmove 1 const 7 Using where; Using filesort
1 SIMPLE s ref game,pilot,sector... game_2 6 const,fightclub_alpha.p.id,const 1 Using where; Using index
edit: I cut some of the unnecessary pieces out of my queries/table structure.
Anybody have any ideas?
the best thing that you can do is to make indexes:
index that covers table ships with fields: game + pilot + sector + hull (in this specific order)
pilots: game + id
this particular query will always use filesort, because it has not range condition p.id <> 2
http://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html
In some cases, MySQL cannot use
indexes to resolve the ORDER BY,
although it still uses indexes to find
the rows that match the WHERE clause.
These cases include the following ...
The key used to fetch the rows is not
the same as the one used in the ORDER
BY