Table Structure:
CREATE TABLE IF NOT EXISTS `newsletters`
(
`id` int(11) NOT NULL auto_increment,
`last_update` int(11) default NULL,
`status` int(11) default '0',
`message_id` varchar(255) default NULL,
PRIMARY KEY (`id`),
KEY `status` (`status`),
KEY `message_id` (`message_id`),
KEY `last_update` (`last_update`)
)
ENGINE=MyISAM DEFAULT CHARSET=latin1;
The Query:
SELECT id, last_update
FROM newsletters
WHERE status = 1
ORDER BY last_update DESC
LIMIT 0, 100
newsletters table has over 3 million records
query takes over 26 seconds to execute
Query explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE newsletters range status status 5 NULL 3043354 Using where; Using filesort
So why is it not using filesort, and how is it a range query?
It's using filesort to sort on last_update. You can avoid the filesort that by changing the index to status, last_update, so MySQL finds all rows with status 1 in the right order.
To further optimize, change the index to status, last_update, id. That allows MySQL to satisfy the query just by looking at the index, without a table lookup.
CREATE INDEX idx_newsletters_status
ON newsletters(status, last_update, id);
Related
As mentioned in the title, I'm migrating from 5.5 to 5.7, and I have 1 query that totally broke:
SELECT
COUNT(*) as count,
detections.vendor
FROM (
SELECT id FROM telemetry_user_agents
WHERE date < DATE_SUB(NOW(), INTERVAL 0 DAY)
AND date > DATE_SUB(NOW(), INTERVAL 7 DAY)
) agents
LEFT JOIN telemetry_detections detections on detections.user_id = agents.id
GROUP BY detections.vendor
ORDER BY count DESC
LIMIT 20
On 5.5 the query takes 4 seconds, on 5.7 it takes 400 seconds (!), with exact same indexes and data. I've read about bugs in 5.7, but I'm pretty sure I can find a workaround with some query optimization and/or better indexes.
user_agents has 200k records, detections has 900k records
Here's the structures:
CREATE TABLE `telemetry_user_agents` (
`id` int(11) NOT NULL,
`date` datetime NOT NULL,
`program` text NOT NULL,
`country` text NOT NULL,
`ip` text NOT NULL,
`operating_system` text NOT NULL,
`bits` int(11) NOT NULL,
`version` text NOT NULL,
`license` text NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
ALTER TABLE `telemetry_user_agents`
ADD PRIMARY KEY (`id`),
ADD KEY `date` (`date`) USING BTREE;
ALTER TABLE `telemetry_user_agents`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
CREATE TABLE `telemetry_detections` (
`id` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`type` text NOT NULL,
`vendor` text NOT NULL,
`name` text NOT NULL,
`value` text NOT NULL,
`hash` text NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
ALTER TABLE `telemetry_detections`
ADD PRIMARY KEY (`id`),
ADD KEY `user_id` (`user_id`);
ALTER TABLE `telemetry_detections`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
And the explains:
5.5
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 95286 Using temporary; Using filesort
1 PRIMARY detections ref user_id user_id 4 agents.id 19
2 DERIVED telemetry_user_agents ALL date NULL NULL NULL 202047 Using where
5.7
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE telemetry_user_agents NULL ALL date NULL NULL NULL 200866 46.50 Using where; Using temporary; Using filesort
1 SIMPLE detections NULL ref user_id user_id 4 adlice_stats.telemetry_user_agents.id 19 100.00 NULL
Any help? :)
EDIT: After I created a bunch of indexes:
5.7:
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE telemetry_user_agents NULL range date,date_2 date_2 5 NULL 131741 100.00 Using where; Using index; Using temporary; Using f...
1 SIMPLE detections NULL ref user_id user_id 4 adlice_stats.telemetry_user_agents.id 19 100.00 NULL
I have a very simple SELECT that resorts to filesort and does not use index.
Consider the following query:
SELECT * FROM forum_topic
WHERE topic_status = 0
ORDER BY modified_date LIMIT 0, 30
on the following table (stripped of a few columns to make it more brief here)
CREATE TABLE `forum_topic` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`slug` varchar(255) NOT NULL,
`forum_id` int(10) NOT NULL DEFAULT '1',
`title` varchar(100) NOT NULL,
`topic_status` tinyint(1) NOT NULL DEFAULT '0',
`post_count` bigint(20) NOT NULL DEFAULT '0',
`modified_date` datetime NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
KEY `slug` (`slug`),
FULLTEXT KEY `title` (`title`),
KEY `modified` (`modified_date`, `topic_status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
EXPLAIN gives the following output
id select_type table? partitions? type? possible_keys? key? key_len? ref? rows? Extra?
1 SIMPLE forum_topic NULL ALL NULL NULL NULL NULL 2075 Using where; Using filesort
Notice how the explain says there are NULL for possible_keys and how it's using filesort after having scanned ALL rows.
Please advice. Thanks.
This query needs topic_status to appear in the most significant position of an index, because it's searching on a constant.
You have
KEY `modified` (`modified_date`, `topic_status`)
and you may want
KEY `mod2` (`topic_status`, `modified_date` )
instead. This may satisfy both the filter and the ORDER BY ... LIMIT part of the query.
Pro tip: Avoid SELECT * and enumerate the columns you actually need instead.
Pro tip: Filesort doesn't necessarily mean what you think it means. It's used anytime MySQL needs to construct an intermediate result set for such things as sorting.
I have the following structure:
CREATE TABLE `tbl1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`tid` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`time` datetime NOT NULL,
`count` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `tid_uid` (`tid`,`user_id`),
KEY `tid_time` (`tid`,`time`)
)
CREATE TABLE `tbl2` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(11) unsigned NOT NULL,
`field_to_order_by` int(11) unsigned NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `user_id` (`user_id`),
KEY `field_to_order_by` (`field_to_order_by`)
)
I'm trying to perform the following query:
SELECT * FROM tbl1
LEFT JOIN tbl2 ON tbl1.user_id=tbl2.user_id
WHERE tbl1.tid=13 AND time > DATE_SUB(NOW(), INTERVAL 1 WEEK)
ORDER BY tbl2.field_to_order_by DESC LIMIT 200
The performance problem I'm facing is due to the ORDER BY of the joined table field. If I remove that or even replace it with a WHERE condition on the same field I'm getting a massive improvement.
How/Can I achieve reasonable performance with this combination of JOIN and ORDER BY or can this only be solved with de-normalization?
This is the EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE tbl1 range tid_uid,tid_time tid_time 12 NULL 221664 Using where; Using temporary; Using filesort
1 SIMPLE tbl2 eq_ref user_id user_id 4 tbl1.user_id 1
I have what seems like a fairly simple table structure, however MySQL is defaulting to a less than optimal index_merge on a simple query.
Here's the table structure:
CREATE TABLE IF NOT EXISTS `event_log` (
`event_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(5) DEFAULT NULL,
`location_id` int(10) DEFAULT NULL,
`object_id` int(5) DEFAULT NULL,
`action_id` int(5) DEFAULT NULL,
`date_event` datetime DEFAULT NULL,
PRIMARY KEY (`event_id`),
KEY `user_id` (`user_id`),
KEY `date_event` (`date_event`),
KEY `action_id` (`action_id`),
KEY `object_id` (`object_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
EXPLAIN on a basic SELECT query
EXPLAIN SELECT date_event
FROM event_log
WHERE user_id =123
AND object_id =456
AND location_id =789
Returns this:
select_type table type possible_keys key key_len ref rows Extra
SIMPLE event_log index_merge user_id,object_id object_id,user_id 5,5 NULL 27 Using intersect(object_id,user_id); Using where
Here's the Extra bit, for easier reading:
Using intersect(object_id,user_id); Using where
Why is MySQL not using standard indexes on this query? Why is it intersecting user_id and object_id?
The most effective index for the query is a composite index that includes all three fields, for example: (object_id, user_id, location_id). Since there is no such index, MySQL does its best to get most of the information from existing indexes.
table:
CREATE TABLE `deal` (
`id` int(11) NOT NULL default '0',
`site` int(11) NOT NULL default '0',
`time` bigint(13) NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `site` (`site`),
KEY `time` (`time`,`site`)
) TYPE=MyISAM
sql query:
select * from `deal` where time>0 && site=8
I create index:time for this query,
but why this query always using index: site?
explain select * from `deal` where time>0 && site=8
output:
table type possible_keys key key_len ref rows Extra
deal ref site,time site 4 const 1 Using where
You need to create composite index site + time (yes, order matters).
So delete both indexes site and time now and create 2:
KEY site (site, time)
KEY time (time)