I'm doing a join between the "favorites" table (3 million rows) the "items" table (600k rows).
The query is taking anywhere from .3 seconds to 2 seconds, and I'm hoping I can optimize it some.
Favorites.faver_profile_id and Items.id are indexed.
Instead of using the faver_profile_id index I created a new index on (faver_profile_id,id), which eliminated the filesort needed when sorting by id. Unfortunately this index doesn't help at all and I'll probably remove it (yay, 3 more hours of downtime to drop the index..)
Any ideas on how I can optimize this query?
In case it helps:
Favorite.removed and Item.removed are "0" 98% of the time.
Favorite.collection_id is NULL about 80% of the time.
SELECT `Item`.`id`, `Item`.`source_image`, `Item`.`cached_image`, `Item`.`source_title`, `Item`.`source_url`, `Item`.`width`, `Item`.`height`, `Item`.`fave_count`, `Item`.`created`
FROM `favorites` AS `Favorite`
LEFT JOIN `items` AS `Item`
ON (`Item`.`removed` = 0 AND `Favorite`.`notice_id` = `Item`.`id`)
WHERE ((`faver_profile_id` = 1) AND (`collection_id` IS NULL) AND (`Favorite`.`removed` = 0) AND (`Item`.`removed` = '0'))
ORDER BY `Favorite`.`id` desc LIMIT 50;
+----+-------------+----------+--------+----------------------------------------------------- ----------+------------------+---------+-----------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+--------+---------------------------------------------------------------+------------------+---------+-----------------------------------------+------+-------------+
| 1 | SIMPLE | Favorite | ref | notice_id,faver_profile_id,collection_id_idx,idx_faver_idx_id | idx_faver_idx_id | 4 | const | 7910 | Using where |
| 1 | SIMPLE | Item | eq_ref | PRIMARY | PRIMARY | 4 | gragland_imgfavebeta.Favorite.notice_id | 1 | Using where |
+----+-------------+----------+--------+---------------------------------------------------------------+------------------+---------+-----------------------------------------+------+-------------+
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| favorites | CREATE TABLE `favorites` (
`id` int(11) NOT NULL auto_increment COMMENT 'unique identifier',
`faver_profile_id` int(11) NOT NULL default '0',
`collection_id` int(11) default NULL,
`collection_order` int(8) default NULL,
`created` datetime NOT NULL default '0000-00-00 00:00:00' COMMENT 'date this record was created',
`modified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP COMMENT 'date this record was modified',
`notice_id` int(11) NOT NULL default '0',
`removed` tinyint(1) NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `notice_id` (`notice_id`),
KEY `faver_profile_id` (`faver_profile_id`),
KEY `collection_id_idx` (`collection_id`),
KEY `idx_faver_idx_id` (`faver_profile_id`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| items |CREATE TABLE `items` (
`id` int(11) NOT NULL auto_increment COMMENT 'unique identifier',
`submitter_id` int(11) NOT NULL default '0' COMMENT 'who made the update',
`source_image` varchar(255) default NULL COMMENT 'update content',
`cached_image` varchar(255) default NULL,
`source_title` varchar(255) NOT NULL default '',
`source_url` text NOT NULL,
`width` int(4) NOT NULL default '0',
`height` int(4) NOT NULL default '0',
`status` varchar(122) NOT NULL default '',
`popular` int(1) NOT NULL default '0',
`made_popular` timestamp NULL default NULL,
`fave_count` int(9) NOT NULL default '0',
`tags` text,
`user_art` tinyint(1) NOT NULL default '0',
`nudity` tinyint(1) NOT NULL default '0',
`created` datetime NOT NULL default '0000-00-00 00:00:00' COMMENT 'date this record was created',
`modified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP COMMENT 'date this record was modified',
`removed` int(1) NOT NULL default '0',
`nofront` tinyint(1) NOT NULL default '0',
`test` varchar(10) NOT NULL default '',
`recs` text,
`recs_data` text,
PRIMARY KEY (`id`),
KEY `notice_profile_id_idx` (`submitter_id`),
KEY `content` (`source_image`),
KEY `idx_popular` (`popular`),
KEY `idx_madepopular` (`made_popular`),
KEY `idx_favecount_idx_id` (`fave_count`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
First of all, you order by favorites.id which is clustered primary key in favorites table. This wil not be necessary of you will join favorites to items instead of items to favorites.
Second, (Item.removed = '0') in WHERE is excess, because the same condition has already been used in JOIN.
Third, change the order of condition in join to:
`Favorite`.`notice_id` = `Item`.`id` AND `Item`.`removed` = 0
the optimizer will be able to use you primary key for index. You may even consider creating (id, removed) index on items table.
Next, create (faver_profile_id, removed) index in favorites (or better update faver_profile_id index) and change the order of conditions in WHERE to the following:
(`faver_profile_id` = 1)
AND (`Favorite`.`removed` = 0)
AND (`collection_id` IS NULL)
UPD: I am sorry, I missed that you already join favorites to items. Then the ORDER BY is not needed. You should result in something like the following:
SELECT
`Item`.`id`,
`Item`.`source_image`,
`Item`.`cached_image`,
`Item`.`source_title`,
`Item`.`source_url`,
`Item`.`width`,
`Item`.`height`,
`Item`.`fave_count`,
`Item`.`created`
FROM `favorites` AS `Favorite`
LEFT JOIN `items` AS `Item`
ON (`Favorite`.`notice_id` = `Item`.`id` AND `Item`.`removed` = 0)
WHERE `faver_profile_id` = 1
AND `Favorite`.`removed` = 0
AND `collection_id` IS NULL
LIMIT 50;
And one more thing, when you have KEY idx_faver_idx_id (faver_profile_id,id) you do not need KEY faver_profile_id (faver_profile_id), because the second index just duplicates half of the idx_faver_idx_id. I hope you will extend the second index, as I suggested.
Get a copy of your table from backup, and try to make an index on Favorite table covering all WHERE and JOIN conditions, namely (removed, collection_id, profile_id). Do the same with Item. It might help, but will make inserts potentially much slower.
The SQL engine won't use an index if it still has to do full table scan due to constraints, would it?
Related
I have a table named 'response_set' with following indexes (result of 'show create table response_set;'):
| response_set | CREATE TABLE `response_set` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`survey_id` int(11) NOT NULL DEFAULT '0',
`respondent_id` int(11) DEFAULT NULL,
`ext_ref` varchar(64) DEFAULT NULL,
`email_addr` varchar(128) DEFAULT NULL,
`ip` varchar(32) DEFAULT NULL,
`t` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`time_taken` int(11) DEFAULT NULL,
`category_id` int(11) DEFAULT NULL,
`duplicate` int(1) DEFAULT '0',
`email_group` varchar(30) DEFAULT NULL,
`external_email_id` int(11) DEFAULT NULL,
`geo_code_country` varchar(64) DEFAULT NULL,
`geo_code_country_code` varchar(2) DEFAULT NULL,
`terminated_survey` int(1) DEFAULT NULL,
`geo_code_region` varchar(128) DEFAULT NULL,
`geo_code_city` varchar(3) DEFAULT NULL,
`geo_code_area_code` varchar(3) DEFAULT NULL,
`geo_code_dma_code` varchar(3) DEFAULT NULL,
`restart_url` varchar(255) DEFAULT NULL,
`inset_list` varchar(1024) DEFAULT NULL,
`custom1` varchar(1024) DEFAULT NULL,
`custom2` varchar(1024) DEFAULT NULL,
`custom3` varchar(1024) DEFAULT NULL,
`custom4` varchar(1024) DEFAULT NULL,
`panel_member_id` int(11) DEFAULT NULL,
`external_id` int(11) DEFAULT NULL,
`weight` float DEFAULT NULL,
`custom5` varchar(1024) DEFAULT NULL,
`quota_overlimit` int(1) DEFAULT '0',
`panel_id` int(11) DEFAULT NULL,
`referer_url` varchar(255) DEFAULT NULL,
`referer_domain` varchar(64) DEFAULT NULL,
`user_agent` varchar(255) DEFAULT NULL,
`longitude` decimal(15,12) DEFAULT '0.000000000000',
`latitude` decimal(15,12) DEFAULT '0.000000000000',
`radius` decimal(7,2) DEFAULT '0.00',
`cx_business_unit_id` int(11) DEFAULT '0',
`survey_link_id` int(11) DEFAULT '0',
`data_quality_flag` int(1) DEFAULT '0',
`data_quality_score` double DEFAULT '0',
`extended_info_json` json DEFAULT NULL,
`updated_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`channel` int(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `panel_member_id` (`panel_member_id`),
KEY `panel_member_id_2` (`panel_member_id`),
KEY `email_group` (`email_group`),
KEY `email_group_2` (`email_group`),
KEY `survey_timestamp_idx` (`survey_id`,`t`),
KEY `cx_business_unit_id_idx` (`cx_business_unit_id`),
KEY `data_quality_flag_idx` (`data_quality_flag`),
KEY `data_quality_score_idx` (`data_quality_score`),
KEY `survey_timestamp_terminated_idx` (`survey_id`,`t`,`terminated_survey`),
KEY `survey_idx` (`survey_id`)
) ENGINE=InnoDB AUTO_INCREMENT=39759 DEFAULT CHARSET=utf8 |
Now I am executing the following query on a page to retrieve the response_set rows based on survey_id and order by id:
SELECT *
FROM response_set a
WHERE a.survey_id = 1602673827
ORDER BY a.id limit 100;
The issue is sometimes the query is taking more than 30 seconds to be executed and this behaviour is inconsistent (as it sometimes happen when order by a.id and sometimes when order by a.id DESC as the user can view the response sets in ascending or descending order on the page) for different survey_id.
There are approx 6.2 million records in the table and for the given survey_id (1602673827) there are 45,800 records. On using the EXPLAIN SELECT statement to understand the query execution plan, I got the following info:
+----+-------------+-------+------------+-------+------------------------------------------------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+------------------------------------------------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | a | NULL | index | survey_timestamp_idx,survey_timestamp_terminated_idx | PRIMARY | 4 | NULL | 6863 | 1.46 | Using where |
+----+-------------+-------+------------+-------+------------------------------------------------------+---------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
Now I am not able to understand that even though the indexes -> 'survey_timestamp_idx,survey_timestamp_terminated_idx' are present, why is MySQL not using the indexes and is opting for the full table scan. Also when i modify the query as follows:
SELECT *
FROM response_set a USE INDEX (survey_timestamp_idx)
WHERE a.survey_id = 1602673827
ORDER BY a.id limit 100;
The query execution time is reduced to 0.17 seconds. On doing the EXPLAIN for the modified query, I get the following info:
+----+-------------+-------+------------+------+----------------------+----------------------+---------+-------+-------+----------+---------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+----------------------+----------------------+---------+-------+-------+----------+---------------------------------------+
| 1 | SIMPLE | a | NULL | ref | survey_timestamp_idx | survey_timestamp_idx | 4 | const | 87790 | 100.00 | Using index condition; Using filesort |
+----+-------------+-------+------------+------+----------------------+----------------------+---------+-------+-------+----------+---------------------------------------+
1 row in set, 1 warning (0.00 sec)
However, I don't want to explicitly use 'USE INDEX' in the query as the where clause is dynamic and may contain following combinations in where clause as per the user's choice of filter:
1. where survey_id = ?;
2. where survey_id = ? and t = ?; (t is timestamp)
3. where survey_id = ? and terminated_survey = ?;
4. where survey_id = ? and t = ? and terminated_survey = ?;
Also, if I remove the ORDER BY clause from the query, the query always uses index and gets executed very fast.
Is there any other way, so that the MySQL query engine chooses the correct (faster) execution plan (by using correct indexes) when ORDER BY clause is present in query?
I am using MySQL version : 5.7.22
I have read the MySQL official documentation for ORDER BY query optimization (https://dev.mysql.com/doc/refman/5.5/en/order-by-optimization.html) and tried adding composite index on (id, survey_id) and (survey_id, id) but it didn't work. Can somebody please help?
where survey_id = ?;
where survey_id = ? and t = ?; (t is timestamp)
where survey_id = ? and terminated_survey = ?;
where survey_id = ? and t = ? and terminated_survey = ?;
Assuming you have ORDER BY id ASC (or DESC), then you need 4 indexes to handle all of them optimally. Start with the 1, 2, or 3 columns (in any order) mentioned in the WHERE, then finish with id.
I cannot explain why KEY survey_idx (survey_id) was not used for the query in question, nor was that index a "possible_key" in the EXPLAIN. It is as if something changed between running the queries and posting this Question. Please recheck.
BTW, INT(1) still takes 4 bytes; you probably wanted the one-byte TINYINT UNSIGNED. Many of the other fields are bigger than necessary. Size plays into performance, at least a little.
0.17s -- Might be even faster with FORCE INDEX(survey_idx)
Starting with the PRIMARY KEY (as in (id, survey_id)) is almost always useless. An index should start things that are tested with =, then move onto something tested as a range or a GROUP BY or (as in your case), ORDER BY.
Cookbook: http://mysql.rjweb.org/doc.php/index_cookbook_mysql
I have table with user statistics. Statistics are generated requesting URL.
Under this URL with GET parameters (event_id) is PHP script. Is response with transparent [1x1] GIF.
event is a child and insertion and insertion is a child from campaign etc up to account.
Where are two types of statistics based on statistic_type.
One account have many spaces (space_id).
This is table CREATE:
CREATE TABLE `statistic` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`statistic_type` TINYINT(3) UNSIGNED NOT NULL,
`account_id` TINYINT(3) UNSIGNED NOT NULL,
`advertiser_id` SMALLINT(5) UNSIGNED NOT NULL,
`campaign_id` SMALLINT(5) UNSIGNED NOT NULL,
`insertion_id` MEDIUMINT(8) UNSIGNED NOT NULL,
`event_id` INT(10) UNSIGNED NOT NULL,
`space_id` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
`date` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`stat_platform_id` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
`stat_browser_id` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
`stat_device_type_id` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
`major_version` TINYINT(4) UNSIGNED NULL DEFAULT NULL,
`uid` CHAR(19) NOT NULL,
`referrer` VARCHAR(255) NULL DEFAULT NULL,
`useragent` VARCHAR(255) NULL DEFAULT NULL,
`ipv4` INT(11) UNSIGNED NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `IDX_uid` (`uid`),
INDEX `Indeks 2` (`statistic_type`, `account_id`, `advertiser_id`, `campaign_id`, `insertion_id`, `event_id`, `space_id`, `date`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPRESSED
AUTO_INCREMENT=427891347;
Now I need to get all and unique (based on uid field) number of rows for campaign_id grouped by statistic_type, event_id, space.
All table have now 428067039 rows. It is still growing about 1500000 rows daily.
Result should have up to 100 rows.
Maximum filtered and letter aggregate (Rows with one campaign_id) row should be up to 5000000.
Now I user this query:
SELECT
`sub`.`account_id`,
`sub`.`advertiser_id`,
`sub`.`campaign_id`,
`sub`.`insertion_id`,
`sub`.`event_id`,
`sub`.`space_id`,
SUM(`sub`.actions) as 'actions',
count(1) as 'unique_actions'
FROM (
SELECT
`statistic`.`account_id`,
`statistic`.`advertiser_id`,
`statistic`.`campaign_id`,
`statistic`.`insertion_id`,
`statistic`.`event_id`,
`statistic`.`space_id`,
count(1) as 'actions'
FROM `statistic`
WHERE
(`statistic`.`statistic_type`=1) AND
(`statistic`.`account_id`=3) AND
(`statistic`.`advertiser_id`=679) AND
(`statistic`.`campaign_id`=4475) AND
(`statistic`.`insertion_id`=26841)
GROUP BY
`statistic`.`statistic_type`,
`statistic`.`account_id`,
`statistic`.`advertiser_id`,
`statistic`.`campaign_id`,
`statistic`.`insertion_id`,
`statistic`.`event_id`,
`statistic`.`space_id`,
`statistic`.`uid`
) sub
GROUP BY
`sub`.`statistic_type`,
`sub`.`account_id`,
`sub`.`advertiser_id`,
`sub`.`campaign_id`,
`sub`.`insertion_id`,
`sub`.`event_id`,
`sub`.`space_id`;
When I have 400000 filtered by insertion_id rows it count's longer than 7-10 minutes.
Explain:
+----+-------------+------------+------+---------------+----------+---------+------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+------+---------------+----------+---------+------+-------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 157521 | Using temporary; Using filesort |
| 2 | DERIVED | statistic | ref | Indeks 2 | Indeks 2 | 9 | | 627090 | Using where; Using temporary; Using filesort |
+----+-------------+------------+------+---------------+----------+---------+------+-------+----------------------------------------------+
2 rows in set (2.61 sec)
Maybe I need change index, maybe add new?
Maybe I need add new col and insert some value when user requesting URL ?
Normally I recommend against having long indexes, but in this case, I suggest an 8-column index with all the columns of sub's GROUP BY, in exactly that order. The one you have ends with date; I don't know if it might be useful for something else. Mine ends with uid.
I expect that would speed up the subquery, but have no impact on the outer query.
An additional thought... Since 5 of the columns are constant, can you get away with not carrying them through to the outer query? This would cut down on the bulk of the temp table of about 157521 rows, thereby possibly speeding up the outer part some.
I have a raspberry PI 3 running MySQL 5.5.57 - this is the only service running on the RPI.
My app makes a key query (below) which takes 5-7 sec to execute on the MySQL server.
I have done a lot to optimize indexes and FK but it really hasn't helped much. When I do an explain I see that it is using temporary and filesort, which I don't really understand.
Are there any configuration tweaks which should be done when running mysql on a RPI. I don't know much about the various buffers...
Is there anything else I should do to optimise the query?
The table has about 30.000 rows and growing...
This is the query:
SELECT SQL_NO_CACHE distinct `photos`.*
FROM `photos`
LEFT OUTER JOIN `facets` ON `photos`.`id` = `facets`.`photo_id`
WHERE (`photos`.`date_taken` <= '2017-08-24')
AND (photos.status != 1 or photos.status is NULL)
ORDER BY photos.date_taken DESC LIMIT 500 OFFSET 500;
This is the table setup:
CREATE TABLE `photos` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`date_taken` datetime DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`file_extension` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`file_size` int(11) DEFAULT NULL,
`location_id` bigint(20) DEFAULT NULL,
`make` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`model` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`original_height` int(11) DEFAULT NULL,
`original_width` int(11) DEFAULT NULL,
`longitude` decimal(16,10) DEFAULT NULL,
`latitude` decimal(16,10) DEFAULT NULL,
`status` int(11) DEFAULT ''0'',
`phash` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`org_id` int(11) DEFAULT NULL,
`lg_id` int(11) DEFAULT NULL,
`md_id` int(11) DEFAULT NULL,
`tm_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_photos_on_location_id` (`location_id`),
KEY `index_photos_on_date_taken` (`date_taken`),
KEY `index_photos_on_status` (`status`),
KEY `index_photos_on_phash` (`phash`),
CONSTRAINT `fk_rails_47f4e5f105` FOREIGN KEY (`location_id`) REFERENCES `locations` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=25672 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
When I do an explain on the query then this is what I get:
+----+-------------+--------+-------+---------------------------------------------------+----------------------------+---------+-------------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+-------+---------------------------------------------------+----------------------------+---------+-------------------+-------+----------------------------------------------+
| 1 | SIMPLE | photos | range | index_photos_on_date_taken,index_photos_on_status | index_photos_on_date_taken | 9 | NULL | 13147 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | facets | ref | index_facets_on_photo_id | index_facets_on_photo_id | 9 | pt_prod.photos.id | 1 | Using index; Distinct |
+----+-------------+--------+-------+---------------------------------------------------+----------------------------+---------+-------------------+-------+----------------------------------------------+
The query will at times be extended to:
SELECT DISTINCT `photos`.*
FROM `photos`
LEFT OUTER JOIN `facets` ON `photos`.`id` = `facets`.`photo_id`
LEFT OUTER JOIN `tags` ON `facets`.`source_id` = `tags`.`id`
LEFT OUTER JOIN `comments` ON `facets`.`source_id` = `comments`.`id`
WHERE `photos`.`date_taken` >= '2017-01-25'
AND `photos`.`date_taken` <= '2018-01-10'
AND `locations`.`country_id` = 16
AND `locations`.`city_id` = 21
OR `facets`.`source_id` = 9 AND `facets`.`type` = 'AlbumFacet'
OR `facets`.`source_id` = 9 AND `facets`.`type` = 'TagFacet'
THIS ANSWERS THE ORIGINAL VERSION OF THE QUESTION.
Your query is only using columns form the first table. I would write it as:
SELECT SQL_NO_CACHE `photos`.*
FROM `photos` p
LEFT OUTER JOIN `facets` ON `photos`.`id` = `facets`.`photo_id`
WHERE (p.`date_taken` <= '2017-08-24') AND (NOT p.status <=> 1) AND
EXISTS (SELECT 1 FROM facets f WHERE pid = f.photo_id)
ORDER BY p.date_taken DESC
LIMIT 500 OFFSET 500;
Removing the SELECT DISTINCT should be a bit win. You should also have an index on facets(photo_id).
An index on (date_taken, status) might help. However, it is not clear how selective your conditions are, so an index on photos might not be of much use.
The following rails query throws back in slow query log:
Class ParserRun
scope :active, -> {
where(completed_at: nil)
.joins('LEFT JOIN system_events ON parser_runs.id = system_events.parser_run_id')
.where("system_events.created_at > '#{active_system_events_threshold}' OR parser_runs.created_at > '#{1.minute.ago.to_s(:db)}'")
}
How can I optimize this?
Slow querylog:
SELECT `parser_runs`.*
FROM `parser_runs`
INNER JOIN `system_events` ON `system_events`.`parser_run_id` = `parser_runs`.`id`
WHERE `parser_runs`.`type` IN ('DatasetParserRun')
AND `parser_runs`.`completed_at` IS NULL
AND (system_events.created_at <= '2017-08-05 04:03:09');
# Time: 170805 5:03:43
Output of 'show create table parser_runs;'
| parser_runs | CREATE TABLE `parser_runs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`customer_id` int(11) DEFAULT NULL,
`options` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`completed_at` datetime DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_parser_runs_on_customer_id` (`customer_id`)
) ENGINE=InnoDB AUTO_INCREMENT=143327 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
output of 'show create table system_events;'
| system_events | CREATE TABLE `system_events` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`log_level` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`customer_id` int(11) DEFAULT NULL,
`classification` int(11) DEFAULT NULL,
`information` text COLLATE utf8_unicode_ci,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`parser_run_id` int(11) DEFAULT NULL,
`notified` tinyint(1) DEFAULT '0',
`dataset_log_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_system_events_on_classification` (`classification`),
KEY `index_system_events_on_customer_id` (`customer_id`),
KEY `index_system_events_on_parser_run_id` (`parser_run_id`),
KEY `index_system_events_on_dataset_log_id` (`dataset_log_id`)
) ENGINE=InnoDB AUTO_INCREMENT=730539 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
Output of EXPLAIN:
EXPLAIN for: SELECT `parser_runs`.* FROM `parser_runs` LEFT JOIN system_events ON parser_runs.id = system_events.parser_run_id WHERE `parser_runs`.`completed_at` IS NULL AND (system_events.created_at > '2017-08-07 10:09:03')
+----+-------------+---------------+--------+------------------------- -------------+---------+---------+--------------------------------------+- -------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+--------+-------------+
| 1 | SIMPLE | system_events | ALL | index_system_events_on_parser_run_id | NULL | NULL | NULL | 655946 | Using where |
| 1 | SIMPLE | parser_runs | eq_ref | PRIMARY | PRIMARY | 4 | ashblood.system_events.parser_run_id | 1 | Using where |
+----+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+--------+-------------+
2 rows in set (0.00 sec)
The first step in the query execution plan (the output of EXPLAIN SELECT ...) indicates that the whole system_events table is being scanned in order to check which rows in the system_events table will be used in the join with the parser_runs table.
Please, add an index on the created_at column in the system_events and repeat the query. Please, check the new execution path to verify whether the whole table is being scanned, or if the new index is being used.
In addition, although probably not the root of the problem, you could add an index on the type and completed_at columns of the table parser_runs. Please, note that I mean an index on both columns (in the given order) instead of an index on each column.
INDEX(type, completed_at)
INDEX(completed_at, type)
INDEX(created_at, parser_run_id)
INDEX(parser_run_id, created_at)
It is not obvious which indexes the Optimizer will prefer; add all of those.
Don't use joins. Instead break the join queries in separate queries and store those data in variables. And later get your desired results from those data.
I've this query, which works fine except it takes a long while (7 seconds, with 40k records in the jobs table, and 700k in the wq table).
I tried an EXPLAIN and it says its looking at all the records in the job table, and not using any of the indexes.
I don't know how to tell MySQL that it should use the jobs.status field to filter the the records before looking up the wq table.
The objective of this, is to get all the records from jobs that have a status != 331, and also any other job which has a wq status of (101, 111, 151).
Query:
SELECT jobs.*
FROM jobs
LEFT OUTER JOIN wq ON (wq.job = jobs.id AND jobs.status IN (341, 331) AND wq.status IN (101, 111, 151))
WHERE ((wq.info is not NULL) or (jobs.status != 331 and ack = 0))
EXPLAIN output:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE jobs ALL ack,status,status_ack NULL NULL NULL 38111 Using filesort
1 SIMPLE wq ref PRIMARY,job,status PRIMARY 4 cts.jobs.id 20 Using where
Table definitions:
CREATE TABLE jobs ( id int(10) NOT NULL AUTO_INCREMENT,
comment varchar(100) NOT NULL DEFAULT '',
profile varchar(60) NOT NULL DEFAULT '',
start_at int(10) NOT NULL DEFAULT '0',
data text NOT NULL,
status int(10) NOT NULL DEFAULT '0',
info varchar(200) NOT NULL DEFAULT '',
finish int(10) NOT NULL DEFAULT '0',
priority int(5) NOT NULL DEFAULT '0',
ack tinyint(4) NOT NULL DEFAULT '0',
PRIMARY KEY (id),
KEY start_at (start_at),
KEY status (status),
KEY status_ack (status,
ack) ) ENGINE=MyISAM AUTO_INCREMENT=2037530 DEFAULT CHARSET=latin1;
CREATE TABLE wq ( job int(10) NOT NULL DEFAULT '0',
process varchar(60) NOT NULL DEFAULT '',
step varchar(60) NOT NULL DEFAULT '',
status int(10) NOT NULL DEFAULT '0',
run_at int(10) NOT NULL DEFAULT '0',
original_run_at int(10) NOT NULL DEFAULT '0',
info varchar(200) NOT NULL DEFAULT '',
pos int(10) NOT NULL DEFAULT '0',
changed_at int(10) NOT NULL DEFAULT '0',
file varchar(60) NOT NULL DEFAULT '',
PRIMARY KEY (job,
process,
step,
file),
KEY job (job),
KEY status (status) ) ENGINE=MyISAM DEFAULT CHARSET=latin1
Unfortunately mysql (and perhaps any dbms) cannot optimize expressions like jobs.status != 331 and ack = 0 because B-Tree is not a structure that allows to find fast anything that is-not-equal-to-a-constant-value. Thus you'll always get a fullscan.
If there were some better condition like jobs.status = 331 and ack = 0 (note on the fact that i've changed != to =) then it would be an advice to speed up this query:
split the query into 2, joined by UNION ALL
replace in one query LEFT JOIN to INNER JOIN (in the one that implies that wq.info is not NULL)