Counting unique rows performance - mysql

I have table with user statistics. Statistics are generated requesting URL.
Under this URL with GET parameters (event_id) is PHP script. Is response with transparent [1x1] GIF.
event is a child and insertion and insertion is a child from campaign etc up to account.
Where are two types of statistics based on statistic_type.
One account have many spaces (space_id).
This is table CREATE:
CREATE TABLE `statistic` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`statistic_type` TINYINT(3) UNSIGNED NOT NULL,
`account_id` TINYINT(3) UNSIGNED NOT NULL,
`advertiser_id` SMALLINT(5) UNSIGNED NOT NULL,
`campaign_id` SMALLINT(5) UNSIGNED NOT NULL,
`insertion_id` MEDIUMINT(8) UNSIGNED NOT NULL,
`event_id` INT(10) UNSIGNED NOT NULL,
`space_id` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
`date` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`stat_platform_id` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
`stat_browser_id` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
`stat_device_type_id` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
`major_version` TINYINT(4) UNSIGNED NULL DEFAULT NULL,
`uid` CHAR(19) NOT NULL,
`referrer` VARCHAR(255) NULL DEFAULT NULL,
`useragent` VARCHAR(255) NULL DEFAULT NULL,
`ipv4` INT(11) UNSIGNED NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `IDX_uid` (`uid`),
INDEX `Indeks 2` (`statistic_type`, `account_id`, `advertiser_id`, `campaign_id`, `insertion_id`, `event_id`, `space_id`, `date`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPRESSED
AUTO_INCREMENT=427891347;
Now I need to get all and unique (based on uid field) number of rows for campaign_id grouped by statistic_type, event_id, space.
All table have now 428067039 rows. It is still growing about 1500000 rows daily.
Result should have up to 100 rows.
Maximum filtered and letter aggregate (Rows with one campaign_id) row should be up to 5000000.
Now I user this query:
SELECT
`sub`.`account_id`,
`sub`.`advertiser_id`,
`sub`.`campaign_id`,
`sub`.`insertion_id`,
`sub`.`event_id`,
`sub`.`space_id`,
SUM(`sub`.actions) as 'actions',
count(1) as 'unique_actions'
FROM (
SELECT
`statistic`.`account_id`,
`statistic`.`advertiser_id`,
`statistic`.`campaign_id`,
`statistic`.`insertion_id`,
`statistic`.`event_id`,
`statistic`.`space_id`,
count(1) as 'actions'
FROM `statistic`
WHERE
(`statistic`.`statistic_type`=1) AND
(`statistic`.`account_id`=3) AND
(`statistic`.`advertiser_id`=679) AND
(`statistic`.`campaign_id`=4475) AND
(`statistic`.`insertion_id`=26841)
GROUP BY
`statistic`.`statistic_type`,
`statistic`.`account_id`,
`statistic`.`advertiser_id`,
`statistic`.`campaign_id`,
`statistic`.`insertion_id`,
`statistic`.`event_id`,
`statistic`.`space_id`,
`statistic`.`uid`
) sub
GROUP BY
`sub`.`statistic_type`,
`sub`.`account_id`,
`sub`.`advertiser_id`,
`sub`.`campaign_id`,
`sub`.`insertion_id`,
`sub`.`event_id`,
`sub`.`space_id`;
When I have 400000 filtered by insertion_id rows it count's longer than 7-10 minutes.
Explain:
+----+-------------+------------+------+---------------+----------+---------+------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+------+---------------+----------+---------+------+-------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 157521 | Using temporary; Using filesort |
| 2 | DERIVED | statistic | ref | Indeks 2 | Indeks 2 | 9 | | 627090 | Using where; Using temporary; Using filesort |
+----+-------------+------------+------+---------------+----------+---------+------+-------+----------------------------------------------+
2 rows in set (2.61 sec)
Maybe I need change index, maybe add new?
Maybe I need add new col and insert some value when user requesting URL ?

Normally I recommend against having long indexes, but in this case, I suggest an 8-column index with all the columns of sub's GROUP BY, in exactly that order. The one you have ends with date; I don't know if it might be useful for something else. Mine ends with uid.
I expect that would speed up the subquery, but have no impact on the outer query.
An additional thought... Since 5 of the columns are constant, can you get away with not carrying them through to the outer query? This would cut down on the bulk of the temp table of about 157521 rows, thereby possibly speeding up the outer part some.

Related

MySQL query with ORDER BY takes long time to execute

I have a table named 'response_set' with following indexes (result of 'show create table response_set;'):
| response_set | CREATE TABLE `response_set` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`survey_id` int(11) NOT NULL DEFAULT '0',
`respondent_id` int(11) DEFAULT NULL,
`ext_ref` varchar(64) DEFAULT NULL,
`email_addr` varchar(128) DEFAULT NULL,
`ip` varchar(32) DEFAULT NULL,
`t` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`time_taken` int(11) DEFAULT NULL,
`category_id` int(11) DEFAULT NULL,
`duplicate` int(1) DEFAULT '0',
`email_group` varchar(30) DEFAULT NULL,
`external_email_id` int(11) DEFAULT NULL,
`geo_code_country` varchar(64) DEFAULT NULL,
`geo_code_country_code` varchar(2) DEFAULT NULL,
`terminated_survey` int(1) DEFAULT NULL,
`geo_code_region` varchar(128) DEFAULT NULL,
`geo_code_city` varchar(3) DEFAULT NULL,
`geo_code_area_code` varchar(3) DEFAULT NULL,
`geo_code_dma_code` varchar(3) DEFAULT NULL,
`restart_url` varchar(255) DEFAULT NULL,
`inset_list` varchar(1024) DEFAULT NULL,
`custom1` varchar(1024) DEFAULT NULL,
`custom2` varchar(1024) DEFAULT NULL,
`custom3` varchar(1024) DEFAULT NULL,
`custom4` varchar(1024) DEFAULT NULL,
`panel_member_id` int(11) DEFAULT NULL,
`external_id` int(11) DEFAULT NULL,
`weight` float DEFAULT NULL,
`custom5` varchar(1024) DEFAULT NULL,
`quota_overlimit` int(1) DEFAULT '0',
`panel_id` int(11) DEFAULT NULL,
`referer_url` varchar(255) DEFAULT NULL,
`referer_domain` varchar(64) DEFAULT NULL,
`user_agent` varchar(255) DEFAULT NULL,
`longitude` decimal(15,12) DEFAULT '0.000000000000',
`latitude` decimal(15,12) DEFAULT '0.000000000000',
`radius` decimal(7,2) DEFAULT '0.00',
`cx_business_unit_id` int(11) DEFAULT '0',
`survey_link_id` int(11) DEFAULT '0',
`data_quality_flag` int(1) DEFAULT '0',
`data_quality_score` double DEFAULT '0',
`extended_info_json` json DEFAULT NULL,
`updated_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`channel` int(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `panel_member_id` (`panel_member_id`),
KEY `panel_member_id_2` (`panel_member_id`),
KEY `email_group` (`email_group`),
KEY `email_group_2` (`email_group`),
KEY `survey_timestamp_idx` (`survey_id`,`t`),
KEY `cx_business_unit_id_idx` (`cx_business_unit_id`),
KEY `data_quality_flag_idx` (`data_quality_flag`),
KEY `data_quality_score_idx` (`data_quality_score`),
KEY `survey_timestamp_terminated_idx` (`survey_id`,`t`,`terminated_survey`),
KEY `survey_idx` (`survey_id`)
) ENGINE=InnoDB AUTO_INCREMENT=39759 DEFAULT CHARSET=utf8 |
Now I am executing the following query on a page to retrieve the response_set rows based on survey_id and order by id:
SELECT *
FROM response_set a
WHERE a.survey_id = 1602673827
ORDER BY a.id limit 100;
The issue is sometimes the query is taking more than 30 seconds to be executed and this behaviour is inconsistent (as it sometimes happen when order by a.id and sometimes when order by a.id DESC as the user can view the response sets in ascending or descending order on the page) for different survey_id.
There are approx 6.2 million records in the table and for the given survey_id (1602673827) there are 45,800 records. On using the EXPLAIN SELECT statement to understand the query execution plan, I got the following info:
+----+-------------+-------+------------+-------+------------------------------------------------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+------------------------------------------------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | a | NULL | index | survey_timestamp_idx,survey_timestamp_terminated_idx | PRIMARY | 4 | NULL | 6863 | 1.46 | Using where |
+----+-------------+-------+------------+-------+------------------------------------------------------+---------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
Now I am not able to understand that even though the indexes -> 'survey_timestamp_idx,survey_timestamp_terminated_idx' are present, why is MySQL not using the indexes and is opting for the full table scan. Also when i modify the query as follows:
SELECT *
FROM response_set a USE INDEX (survey_timestamp_idx)
WHERE a.survey_id = 1602673827
ORDER BY a.id limit 100;
The query execution time is reduced to 0.17 seconds. On doing the EXPLAIN for the modified query, I get the following info:
+----+-------------+-------+------------+------+----------------------+----------------------+---------+-------+-------+----------+---------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+----------------------+----------------------+---------+-------+-------+----------+---------------------------------------+
| 1 | SIMPLE | a | NULL | ref | survey_timestamp_idx | survey_timestamp_idx | 4 | const | 87790 | 100.00 | Using index condition; Using filesort |
+----+-------------+-------+------------+------+----------------------+----------------------+---------+-------+-------+----------+---------------------------------------+
1 row in set, 1 warning (0.00 sec)
However, I don't want to explicitly use 'USE INDEX' in the query as the where clause is dynamic and may contain following combinations in where clause as per the user's choice of filter:
1. where survey_id = ?;
2. where survey_id = ? and t = ?; (t is timestamp)
3. where survey_id = ? and terminated_survey = ?;
4. where survey_id = ? and t = ? and terminated_survey = ?;
Also, if I remove the ORDER BY clause from the query, the query always uses index and gets executed very fast.
Is there any other way, so that the MySQL query engine chooses the correct (faster) execution plan (by using correct indexes) when ORDER BY clause is present in query?
I am using MySQL version : 5.7.22
I have read the MySQL official documentation for ORDER BY query optimization (https://dev.mysql.com/doc/refman/5.5/en/order-by-optimization.html) and tried adding composite index on (id, survey_id) and (survey_id, id) but it didn't work. Can somebody please help?
where survey_id = ?;
where survey_id = ? and t = ?; (t is timestamp)
where survey_id = ? and terminated_survey = ?;
where survey_id = ? and t = ? and terminated_survey = ?;
Assuming you have ORDER BY id ASC (or DESC), then you need 4 indexes to handle all of them optimally. Start with the 1, 2, or 3 columns (in any order) mentioned in the WHERE, then finish with id.
I cannot explain why KEY survey_idx (survey_id) was not used for the query in question, nor was that index a "possible_key" in the EXPLAIN. It is as if something changed between running the queries and posting this Question. Please recheck.
BTW, INT(1) still takes 4 bytes; you probably wanted the one-byte TINYINT UNSIGNED. Many of the other fields are bigger than necessary. Size plays into performance, at least a little.
0.17s -- Might be even faster with FORCE INDEX(survey_idx)
Starting with the PRIMARY KEY (as in (id, survey_id)) is almost always useless. An index should start things that are tested with =, then move onto something tested as a range or a GROUP BY or (as in your case), ORDER BY.
Cookbook: http://mysql.rjweb.org/doc.php/index_cookbook_mysql

MySQL doesn't use entire index when combining join and range

I'm trying to optimize a simple query that joins between two tables and applies a range condition.
From the explain plan below, you can see that the index inv_quantity_on_hand is only used partially (4 bytes, only for the first column - inv_item_sk). I would expect the entire index to be used, as the second part (inv_quantity_on_hand) of the index is used in the WHERE clause in a range condition.
Please note that this happens only with a join and a range condition. Replacing the range condition to a constant equality comparison (inv_quantity_on_hand = 5) will change the explain plan and MySQL will use the entire index.
It seems to be an instance of this bug: https://bugs.mysql.com/bug.php?id=8569.
I checked it with MySQL 5.7 and it still happens. Anyone can think of a good workaround please?
Schema structure:
CREATE TABLE `inventory` (
`inv_date_sk` INT(11) NOT NULL,
`inv_item_sk` INT(11) NOT NULL,
`inv_warehouse_sk` INT(11) NOT NULL,
`inv_quantity_on_hand` INT(11) DEFAULT NULL,
PRIMARY KEY (`inv_date_sk` , `inv_item_sk` , `inv_warehouse_sk`),
KEY `inv_w` (`inv_warehouse_sk`),
KEY `inv_i` (`inv_item_sk`),
KEY `inv_quantity_on_hand_index` (`inv_item_sk` , `inv_quantity_on_hand`),
CONSTRAINT `inv_d` FOREIGN KEY (`inv_date_sk`)
REFERENCES `date_dim` (`d_date_sk`)
ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `inv_i` FOREIGN KEY (`inv_item_sk`)
REFERENCES `item` (`i_item_sk`)
ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `inv_w` FOREIGN KEY (`inv_warehouse_sk`)
REFERENCES `warehouse` (`w_warehouse_sk`)
ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=INNODB DEFAULT CHARSET=UTF8
CREATE TABLE `item` (
`i_item_sk` INT(11) NOT NULL,
`i_item_id` CHAR(16) NOT NULL,
`i_rec_start_date` DATE DEFAULT NULL,
`i_rec_end_date` DATE DEFAULT NULL,
`i_item_desc` VARCHAR(200) DEFAULT NULL,
`i_current_price` DECIMAL(7 , 2 ) DEFAULT NULL,
`i_wholesale_cost` DECIMAL(7 , 2 ) DEFAULT NULL,
`i_brand_id` INT(11) DEFAULT NULL,
`i_brand` CHAR(50) DEFAULT NULL,
`i_class_id` INT(11) DEFAULT NULL,
`i_class` CHAR(50) DEFAULT NULL,
`i_category_id` INT(11) DEFAULT NULL,
`i_category` CHAR(50) DEFAULT NULL,
`i_manufact_id` INT(11) DEFAULT NULL,
`i_manufact` CHAR(50) DEFAULT NULL,
`i_size` CHAR(20) DEFAULT NULL,
`i_formulation` CHAR(20) DEFAULT NULL,
`i_color` CHAR(20) DEFAULT NULL,
`i_units` CHAR(10) DEFAULT NULL,
`i_container` CHAR(10) DEFAULT NULL,
`i_manager_id` INT(11) DEFAULT NULL,
`i_product_name` CHAR(50) DEFAULT NULL,
PRIMARY KEY (`i_item_sk`),
KEY `item_color_index` (`i_color`)
) ENGINE=INNODB DEFAULT CHARSET=UTF8
Query:
SELECT
*
FROM
inventory
INNER JOIN
item ON inventory.inv_item_sk = item.i_item_sk
WHERE
inventory.inv_quantity_on_hand > 100
AND item.i_color = 'red';
Execution plan:
# id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra
-----+-------------+-----------+------------+------+----------------------------------+----------------------------+---------+----------------------+-----------------+-------------------------
1 | SIMPLE | item | | ref | PRIMARY,item_color_index | item_color_index | 61 | const | 384 | 100.00 |
1 | SIMPLE | inventory | | ref | inv_i,inv_quantity_on_hand_index | inv_quantity_on_hand_index | 4 | tpcds.item.i_item_sk | 615 | 33.33 | Using where; Using index
The multi column indexes are stored as concats of the different columns.
I think MySQL will not evaluate a substring within a multi column index for comparison. When you use inv_quantity_on_hand = 5 (or in(1,2,3,4,5)) MySQL will build the strings from your input for comparison so it can use the full index. Using between or > basically provides an unlimited number of possible substrings to compare (before checking data type). Building all those strings and comparing them would take much more time than using the index for the first column (on-clause) and then check the inv_quantity_on_hand "using where".
use BETWEEN condition instead of conditional operator

Slow Query on Rails joins

The following rails query throws back in slow query log:
Class ParserRun
scope :active, -> {
where(completed_at: nil)
.joins('LEFT JOIN system_events ON parser_runs.id = system_events.parser_run_id')
.where("system_events.created_at > '#{active_system_events_threshold}' OR parser_runs.created_at > '#{1.minute.ago.to_s(:db)}'")
}
How can I optimize this?
Slow querylog:
SELECT `parser_runs`.*
FROM `parser_runs`
INNER JOIN `system_events` ON `system_events`.`parser_run_id` = `parser_runs`.`id`
WHERE `parser_runs`.`type` IN ('DatasetParserRun')
AND `parser_runs`.`completed_at` IS NULL
AND (system_events.created_at <= '2017-08-05 04:03:09');
# Time: 170805 5:03:43
Output of 'show create table parser_runs;'
| parser_runs | CREATE TABLE `parser_runs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`customer_id` int(11) DEFAULT NULL,
`options` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`completed_at` datetime DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_parser_runs_on_customer_id` (`customer_id`)
) ENGINE=InnoDB AUTO_INCREMENT=143327 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
output of 'show create table system_events;'
| system_events | CREATE TABLE `system_events` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`log_level` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`customer_id` int(11) DEFAULT NULL,
`classification` int(11) DEFAULT NULL,
`information` text COLLATE utf8_unicode_ci,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`parser_run_id` int(11) DEFAULT NULL,
`notified` tinyint(1) DEFAULT '0',
`dataset_log_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_system_events_on_classification` (`classification`),
KEY `index_system_events_on_customer_id` (`customer_id`),
KEY `index_system_events_on_parser_run_id` (`parser_run_id`),
KEY `index_system_events_on_dataset_log_id` (`dataset_log_id`)
) ENGINE=InnoDB AUTO_INCREMENT=730539 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
Output of EXPLAIN:
EXPLAIN for: SELECT `parser_runs`.* FROM `parser_runs` LEFT JOIN system_events ON parser_runs.id = system_events.parser_run_id WHERE `parser_runs`.`completed_at` IS NULL AND (system_events.created_at > '2017-08-07 10:09:03')
+----+-------------+---------------+--------+------------------------- -------------+---------+---------+--------------------------------------+- -------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+--------+-------------+
| 1 | SIMPLE | system_events | ALL | index_system_events_on_parser_run_id | NULL | NULL | NULL | 655946 | Using where |
| 1 | SIMPLE | parser_runs | eq_ref | PRIMARY | PRIMARY | 4 | ashblood.system_events.parser_run_id | 1 | Using where |
+----+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+--------+-------------+
2 rows in set (0.00 sec)
The first step in the query execution plan (the output of EXPLAIN SELECT ...) indicates that the whole system_events table is being scanned in order to check which rows in the system_events table will be used in the join with the parser_runs table.
Please, add an index on the created_at column in the system_events and repeat the query. Please, check the new execution path to verify whether the whole table is being scanned, or if the new index is being used.
In addition, although probably not the root of the problem, you could add an index on the type and completed_at columns of the table parser_runs. Please, note that I mean an index on both columns (in the given order) instead of an index on each column.
INDEX(type, completed_at)
INDEX(completed_at, type)
INDEX(created_at, parser_run_id)
INDEX(parser_run_id, created_at)
It is not obvious which indexes the Optimizer will prefer; add all of those.
Don't use joins. Instead break the join queries in separate queries and store those data in variables. And later get your desired results from those data.

MYSQL, very slow order by

I have got two tables. One is a User table with a primary key on the userid and the other table references the user table with a foreign key.
The User table has only one entry (for now) and the other table has one million entrys.
The following join drives me mad:
SELECT p0_.*, p1_.*
FROM photo p0_, User p1_
WHERE p0_.user_id = p1_.user_id
ORDER BY p0_.uploaddate DESC Limit 10 OFFSET 100000
The query takes 12sec on a very fast machine with the order by and 0.0005 sec without the order by.
I've got an index on user_id (IDX_14B78418A76ED395) and a composite index ("search2") on user_id and uploaddate.
EXPLAIN shows the following:
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
| 1 | SIMPLE | p1_ | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | p0_ | ref | IDX_14B78418A76ED395,search2 | IDX_14B78418A76ED395 | 4 | odsfoto.p1_.user_id | 58520 | |
+----+-------------+-------+------+------------------------------+----------------------+---------+---------------------+-------+---------------------------------+
Table definitions:
CREATE TABLE `photo` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`album_id` int(11) DEFAULT NULL,
`exif_id` int(11) DEFAULT NULL,
`title` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`width` int(11) NOT NULL,
`height` int(11) NOT NULL,
`uploaddate` datetime NOT NULL,
`filesize` int(11) DEFAULT NULL,
`path` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`originalFilename` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`mimeType` varchar(200) COLLATE utf8_unicode_ci NOT NULL,
`description` longtext COLLATE utf8_unicode_ci,
`gpsData_id` int(11) DEFAULT NULL,
`views` int(11) DEFAULT NULL,
`likes` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_14B78418B0FC9251` (`exif_id`),
UNIQUE KEY `UNIQ_14B7841867E96507` (`gpsData_id`),
KEY `IDX_14B78418A76ED395` (`user_id`),
KEY `IDX_14B784181137ABCF` (`album_id`),
KEY `search_idx` (`uploaddate`),
KEY `search2` (`user_id`,`uploaddate`),
KEY `search3` (`uploaddate`,`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `user` (
`user_id` int(11) NOT NULL,
`photoCount` int(11) NOT NULL,
`photoViews` int(11) NOT NULL,
`photoComments` int(11) NOT NULL,
`photoLikes` int(11) NOT NULL,
`username` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
What can I do to speed up this query?
Seems you're suffering from MySQL's inability to do late row lookups:
MySQL ORDER BY / LIMIT performance: late row lookups
Late row lookups: InnoDB
Try this:
SELECT p.*, u.*
FROM (
SELECT id
FROM photo
ORDER BY
uploaddate DESC, id DESC
LIMIT 10
OFFSET 100000
) pi
JOIN photo p
ON p.id = pi.id
JOIN user u
ON u.user_id = p.user_id
You need a separate index on uploaddate. This sort will take advantage of composite index only if uploaddate is first column in it.
You can also try to add user_id to ORDER BY:
....
ORDER BY p0_.user_id, p0_.uploaddate
You have two problems:
You need to create an INDEX(user_id, uploaddate) which will greatly increase the efficiency of the query.
You need to find a workaround to using LIMIT 10 OFFSET 100000. MySQL is creating a recordset with 100,000 records in it, then it pulls the last 10 records off the end... that is extremely inefficient.
https://www.percona.com/blog/2006/09/01/mysql-order-by-limit-performance-optimization/
First try to get result based on primary key with out join and use result to query result again.
For ex:
$userIds=mysql::select("select user_id from photo ORDER BY p0_.uploaddate DESC Limit 10 OFFSET 100000");
$photoData=mysql::select("SELECT p0_., p1_.
FROM photo p0_, User p1_
WHERE p0_.user_id = p1_.user_id and p0_.user_id in ($userIds->user_id) order by p0_.uploaddate");
Here we had divided the statement into two parts:
1.We can easily order and get based on primary key and also there are no joins.
2.Getting query results based on id and order by is only on limited columns we can retrieve data in less time
From 30 seconds to 0.015 sec / 0.000 sec using Quassnoi answer !
This is what I called MySql expertise !
I cut out one Join from my personal project (no join with itself)
Select ser.id_table, ser.id_rec, ser.relevance, cnt, title, description, sell_url, medium_thumb,
unique_id_supplier, keywords width, height, media_type
from (
Select ser.id_rec, ser.id_table, ser.relevance, ser.cnt
from searchEngineResults ser
where thisSearch = 16287
order by ser.relevance desc, cnt desc, id_rec
) ser
join photo_resell sou on sou.id = ser.id_rec
#join searchEngineResults ser on ser.id_rec = tmp.id_rec
limit 0, 9

Need help optimizing MYSQL query with join

I'm doing a join between the "favorites" table (3 million rows) the "items" table (600k rows).
The query is taking anywhere from .3 seconds to 2 seconds, and I'm hoping I can optimize it some.
Favorites.faver_profile_id and Items.id are indexed.
Instead of using the faver_profile_id index I created a new index on (faver_profile_id,id), which eliminated the filesort needed when sorting by id. Unfortunately this index doesn't help at all and I'll probably remove it (yay, 3 more hours of downtime to drop the index..)
Any ideas on how I can optimize this query?
In case it helps:
Favorite.removed and Item.removed are "0" 98% of the time.
Favorite.collection_id is NULL about 80% of the time.
SELECT `Item`.`id`, `Item`.`source_image`, `Item`.`cached_image`, `Item`.`source_title`, `Item`.`source_url`, `Item`.`width`, `Item`.`height`, `Item`.`fave_count`, `Item`.`created`
FROM `favorites` AS `Favorite`
LEFT JOIN `items` AS `Item`
ON (`Item`.`removed` = 0 AND `Favorite`.`notice_id` = `Item`.`id`)
WHERE ((`faver_profile_id` = 1) AND (`collection_id` IS NULL) AND (`Favorite`.`removed` = 0) AND (`Item`.`removed` = '0'))
ORDER BY `Favorite`.`id` desc LIMIT 50;
+----+-------------+----------+--------+----------------------------------------------------- ----------+------------------+---------+-----------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+--------+---------------------------------------------------------------+------------------+---------+-----------------------------------------+------+-------------+
| 1 | SIMPLE | Favorite | ref | notice_id,faver_profile_id,collection_id_idx,idx_faver_idx_id | idx_faver_idx_id | 4 | const | 7910 | Using where |
| 1 | SIMPLE | Item | eq_ref | PRIMARY | PRIMARY | 4 | gragland_imgfavebeta.Favorite.notice_id | 1 | Using where |
+----+-------------+----------+--------+---------------------------------------------------------------+------------------+---------+-----------------------------------------+------+-------------+
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| favorites | CREATE TABLE `favorites` (
`id` int(11) NOT NULL auto_increment COMMENT 'unique identifier',
`faver_profile_id` int(11) NOT NULL default '0',
`collection_id` int(11) default NULL,
`collection_order` int(8) default NULL,
`created` datetime NOT NULL default '0000-00-00 00:00:00' COMMENT 'date this record was created',
`modified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP COMMENT 'date this record was modified',
`notice_id` int(11) NOT NULL default '0',
`removed` tinyint(1) NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `notice_id` (`notice_id`),
KEY `faver_profile_id` (`faver_profile_id`),
KEY `collection_id_idx` (`collection_id`),
KEY `idx_faver_idx_id` (`faver_profile_id`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| items |CREATE TABLE `items` (
`id` int(11) NOT NULL auto_increment COMMENT 'unique identifier',
`submitter_id` int(11) NOT NULL default '0' COMMENT 'who made the update',
`source_image` varchar(255) default NULL COMMENT 'update content',
`cached_image` varchar(255) default NULL,
`source_title` varchar(255) NOT NULL default '',
`source_url` text NOT NULL,
`width` int(4) NOT NULL default '0',
`height` int(4) NOT NULL default '0',
`status` varchar(122) NOT NULL default '',
`popular` int(1) NOT NULL default '0',
`made_popular` timestamp NULL default NULL,
`fave_count` int(9) NOT NULL default '0',
`tags` text,
`user_art` tinyint(1) NOT NULL default '0',
`nudity` tinyint(1) NOT NULL default '0',
`created` datetime NOT NULL default '0000-00-00 00:00:00' COMMENT 'date this record was created',
`modified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP COMMENT 'date this record was modified',
`removed` int(1) NOT NULL default '0',
`nofront` tinyint(1) NOT NULL default '0',
`test` varchar(10) NOT NULL default '',
`recs` text,
`recs_data` text,
PRIMARY KEY (`id`),
KEY `notice_profile_id_idx` (`submitter_id`),
KEY `content` (`source_image`),
KEY `idx_popular` (`popular`),
KEY `idx_madepopular` (`made_popular`),
KEY `idx_favecount_idx_id` (`fave_count`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
First of all, you order by favorites.id which is clustered primary key in favorites table. This wil not be necessary of you will join favorites to items instead of items to favorites.
Second, (Item.removed = '0') in WHERE is excess, because the same condition has already been used in JOIN.
Third, change the order of condition in join to:
`Favorite`.`notice_id` = `Item`.`id` AND `Item`.`removed` = 0
the optimizer will be able to use you primary key for index. You may even consider creating (id, removed) index on items table.
Next, create (faver_profile_id, removed) index in favorites (or better update faver_profile_id index) and change the order of conditions in WHERE to the following:
(`faver_profile_id` = 1)
AND (`Favorite`.`removed` = 0)
AND (`collection_id` IS NULL)
UPD: I am sorry, I missed that you already join favorites to items. Then the ORDER BY is not needed. You should result in something like the following:
SELECT
`Item`.`id`,
`Item`.`source_image`,
`Item`.`cached_image`,
`Item`.`source_title`,
`Item`.`source_url`,
`Item`.`width`,
`Item`.`height`,
`Item`.`fave_count`,
`Item`.`created`
FROM `favorites` AS `Favorite`
LEFT JOIN `items` AS `Item`
ON (`Favorite`.`notice_id` = `Item`.`id` AND `Item`.`removed` = 0)
WHERE `faver_profile_id` = 1
AND `Favorite`.`removed` = 0
AND `collection_id` IS NULL
LIMIT 50;
And one more thing, when you have KEY idx_faver_idx_id (faver_profile_id,id) you do not need KEY faver_profile_id (faver_profile_id), because the second index just duplicates half of the idx_faver_idx_id. I hope you will extend the second index, as I suggested.
Get a copy of your table from backup, and try to make an index on Favorite table covering all WHERE and JOIN conditions, namely (removed, collection_id, profile_id). Do the same with Item. It might help, but will make inserts potentially much slower.
The SQL engine won't use an index if it still has to do full table scan due to constraints, would it?