I have a table with nested children. I'm trying to fetch a list of parents sorted by the most recent child, when available, otherwise the parent's created date. My query seemed to work at first, but as I started importing more and more records (#13.6K atm), performance has become a problem.
Version: 10.5.5-MariaDB
Table structure (excluded fields for brevity):
CREATE TABLE `emails` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`created_at` timestamp NULL DEFAULT NULL,
`_lft` int(10) unsigned NOT NULL DEFAULT 0,
`_rgt` int(10) unsigned NOT NULL DEFAULT 0,
`parent_id` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `emails__lft__rgt_parent_id_index` (`_lft`,`_rgt`,`parent_id`) USING BTREE,
KEY `emails__lft__rgt_created_at_index` (`_lft`,`_rgt`,`created_at`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=13484 DEFAULT CHARSET=utf8
Here's the query I'm working with (#21s):
SELECT
`emails`.`id`,
(
SELECT MAX(`descendants`.`created_at`) AS `created_at`
FROM `emails` AS `descendants`
WHERE `descendants`.`_lft` >= `emails`.`_lft`
AND `descendants`.`_rgt` <= `emails`.`_rgt`
) `descendants_created_at`
FROM `emails`
WHERE `parent_id` IS NULL
ORDER BY `descendants_created_at` DESC
LIMIT 25 OFFSET 0;
The _lft and _rgt fields are provided by the lazychaser/laravel-nestedset package and are essentially giving me the descendants for each of the records returned in the main query. It includes the parent as well, so a created_at value is always returned.
Sample output:
| id | created_at | descendants_created_at |
|-------|---------------------|------------------------|
| 13483 | 2021-07-22 12:35:55 | 2021-07-22 12:35:55 |
| 8460 | 2021-04-29 12:56:57 | 2021-07-22 12:35:00 |
| 13481 | 2021-07-22 12:33:22 | 2021-07-22 12:33:22 |
| 3514 | 2021-01-16 09:43:42 | 2021-07-22 12:23:28 |
| 13479 | 2021-07-22 11:28:07 | 2021-07-22 11:28:07 |
| 13478 | 2021-07-22 11:27:09 | 2021-07-22 11:27:09 |
| 13407 | 2021-07-21 10:05:41 | 2021-07-22 10:21:14 |
| 13408 | 2021-07-21 10:05:41 | 2021-07-22 10:21:14 |
| 13389 | 2021-07-21 08:17:23 | 2021-07-22 10:21:14 |
| 13303 | 2021-07-19 14:25:38 | 2021-07-22 10:21:14 |
The problem seems to be once I'm doing the actual ordering here:
ORDER BY `descendants_created_at` DESC
My EXPLAIN looks like this:
UPDATE #1 - Using a LEFT JOIN & adding a parent_id key, this query is now #10s which is better, but still not great:
https://dbfiddle.uk/?rdbms=mariadb_10.5&fiddle=f5442fdfba119cc750c09a19024ccf7c
Related
I am trying to convert some data from old tables to a new structure, where I need to convert a single key ID to composite one, amd it is giving me some trouble:
My table (simplified):
CREATE TABLE `tmplt_spoergsmaal` (
`SpID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`lbnr` int(10) unsigned DEFAULT NULL,
`SpTekst` text,
`SpTitel` varchar(100) NOT NULL DEFAULT '',
`fk_Naeste_Sp` int(10) unsigned DEFAULT NULL,
`kontrol` tinyint(3) unsigned NOT NULL DEFAULT 0,
`kontrol_kilde` int(10) unsigned NOT NULL DEFAULT 0,
FOREIGN KEY (kontrol_kilde)
REFERENCES tmplt_spoergsmaal(SpID)
ON DELETE ignore
)
Sample data:
Note that SPID is sequential
+--------+--------+-----------+-----------+------------+-----------------+
| SpID | lbnr | SpTekst | SpTitel | kontrol | kontrol_kilde |
+--------+--------+-----------+-----------+------------+-----------------+
| 9000 | 100 | blablabla | title1 | 0 | null |
+--------+--------+-----------+-----------+------------+-----------------+
| 9001 | 101 | blablabla | title2 | 0 | null |
+--------+--------+-----------+-----------+------------+-----------------+
| 9002 | 102 | blablabla | title3 | 0 | null |
+--------+--------+-----------+-----------+------------+-----------------+
| 9003 | 103 | blablabla | title4 | 1 | 9000 |
+--------+--------+-----------+-----------+------------+-----------------+
| 9004 | 104 | blablabla | title5 | 1 | 9001 |
+--------+--------+-----------+-----------+------------+-----------------+
I am redesigning the database, and using the lbnr column instead of the kontrol_kilde column. My preliminary query is this:
SELECT spid, lbnr, kontrol, kontrol_kilde, (lbnr- (spid - kontrol_kilde)* kontrol)* kontrol AS k
FROM tmplt_spoergsmaal;
This solves my issue, but an issue cropped up at one point (because of a flip of the subtraction (spid - kontrol_kilde had become kontrol_kilde - spid) which made part of the equation negative. Since the column is unsigned,this caused an error:
Error Code: 1690. BIGINT UNSIGNED value is out of range in
My question:
Can I "cast" the columns in the alias column k so that it is an int instead of unsigned int ?
Well, you can cast() as signed:
SELECT spid, lbnr, kontrol, kontrol_kilde,
cast(lbnr - (spid - kontrol_kilde) * kontrol)* kontrol as signed) AS k
FROM tmplt_spoergsmaal;
I have a question on join two table
TWD_CSD_NEWS_DETAIL (200 million row):
+-----------+------------+--------------+------------------+
| CSD_ID | CSD_ID_DRI | CSD_PARTY_ID | CSD_PARTY_AMOUNT |
+-----------+------------+--------------+------------------+
| 1 | 1 | 1183 | 27870 |
+-----------+------------+--------------+------------------+
| 2 | 1 | 1723 | 12 |
+-----------+------------+--------------+------------------+
| 3 | 1 | 1243 | 87474 |
+-----------+------------+--------------+------------------+
.
.
.
+-----------+------------+--------------+------------------+
| 18575622 | 8881 | 1183 | 27870 |
+-----------+------------+--------------+------------------+
the result of SHOW CREATE TABLE TWD_CSD_NEWS_DETAIL:
CREATE TABLE `TWD_CSD_NEWS_DETAIL` (
`CSD_ID` int(11) NOT NULL AUTO_INCREMENT,
`CSD_ID_CREATED_BY` int(11) DEFAULT NULL,
`CSD_DT_CREATED` datetime DEFAULT NULL,
`CSD_DT_UPD` datetime DEFAULT NULL,
`CSD_ID_DRI` int(11) DEFAULT NULL,
`CSD_ID_UPD_BY` int(11) DEFAULT NULL,
`CSD_PARTY_ID` int(11) DEFAULT NULL,
`CSD_AMOUNT` decimal(26,0) DEFAULT NULL,
`CSD_TIMESTAMP` datetime DEFAULT NULL,
PRIMARY KEY (`CSD_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=184035984 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
TWD_DRI_NEWS_RESULT_HEADER (1 million row) :
+--------+---------------------+----------------+
| DRI_ID | DRI_DATE | DRI_SYM_SYMBOL |
+--------+---------------------+----------------+
| 1 | 2011-11-08 00:00:00 | 1 |
+--------+---------------------+----------------+
| 2 | 2011-11-08 00:00:00 | 2 |
+--------+---------------------+----------------+
| 3 | 2011-11-08 00:00:00 | 3 |
+--------+---------------------+----------------+
.
.
+--------+---------------------+----------------+
| 10001 | 2011-11-11 00:00:00 | 8881 |
+--------+---------------------+----------------+
the result of SHOW CREATE TABLE TWD_DRI_NEWS_RESULT_HEADER :
CREATE TABLE `TWD_DRI_NEWS_RESULT_HEADER` (
`DRI_ID` int(11) NOT NULL AUTO_INCREMENT,
`DRI_DATE` datetime DEFAULT NULL,
`DRI_SYM_SYMBOL` int(11) DEFAULT NULL,
`DRI_TIMESTAMP` datetime DEFAULT NULL,
PRIMARY KEY (`DRI_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=1592193 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
I try to join them with following sql, it works but it will take very long to completed this query when i keep adding csd_id range in where cluase
SELECT
csd.CSD_ID, csd.CSD_ID_DRI, csd.CSD_PARTY_ID, csd.CSD_AMOUNT , dri.DRI_DATE, dri.DRI_SYM_TICKER
FROM
TWD_CSD_NEWS_DETAIL csd
LEFT JOIN
TWD_DRI_NEWS_RESULT_HEADER dri ON dri.DRI_ID = csd.CSD_ID_DRI
WHERE
(
(
( csd_id between 1 and 426029)
|| ( csd_id between 426030 and 851977)
|| ( csd_id between 851978 and 1277890)
..
...
...
)
AND dri.DRI_SYM_SYMBOL = 1
)
do i need create another view to contain result or any faster method to query this? i tried with the range between 1 and 200000000 ther duration and fetch time require 0.197 seconds / 26 seconds
Have you tried using a single range for your query?
SELECT
csd.CSD_ID, csd.CSD_ID_DRI, csd.CSD_PARTY_ID, csd.CSD_SHAREHOLDING ,
dri.DRI_SHAREHOLDING_DATE, dri.DRI_SYM_TICKER
FROM
TWD_CSD_NEWS_DETAIL csd
LEFT JOIN
TWD_DRI_NEWS_RESULT_HEADER dri ON dri.DRI_ID = csd.CSD_ID_DRI
WHERE
csd_id between (1 and 1277890) AND dri.DRI_SYM_SYMBOL = 1;
I do not understand the difference (line 2) of those two EXPLAINs. Maybe someone has a hint for me why mysql acts so different on those, which heavily affects query speed.
The slow query lasts 12 seconds (which equals querying all rows with that query) and uses a join on integer columns while the joined table has just 3 records:
SELECT `inv_assets`.`id` AS `id`, `site`.`description` AS `sitename`,
(SELECT COALESCE(DATE_FORMAT(CONVERT_TZ(MIN(inspdate),'UTC','Europe/Vienna'),'%Y-%m-%d'),'')
FROM `mobuto_inv_inspections` AS `nextinsp`
WHERE ((`nextinsp`.`objectlink` = `inv_assets`.`id`
AND `nextinsp`.`inspdate` >= NOW()))
) AS `nextinsp`
FROM `mobuto_inv_assets` AS `inv_assets`
LEFT JOIN `mobuto_inv_sites` AS `site`
ON (`site`.`siteid` = `inv_assets`.`site`
AND `site`.`_state` IN (2,0))
ORDER BY `inv_assets`.`type` ASC LIMIT 0, 20;
+----+--------------------+------------+--------+----------------+---------+---------+------------------------------+-------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+--------+----------------+---------+---------+------------------------------+-------+----------------------------------------------------+
| 1 | PRIMARY | inv_assets | ALL | NULL | NULL | NULL | NULL | 24857 | Using temporary; Using filesort |
| 1 | PRIMARY | site | ALL | PRIMARY,_state | NULL | NULL | NULL | 3 | Using where; Using join buffer (Block Nested Loop) |
| 2 | DEPENDENT SUBQUERY | nextinsp | ALL | inspdate | NULL | NULL | NULL | 915 | Using where |
+----+--------------------+------------+--------+----------------+---------+---------+------------------------------+-------+----------------------------------------------------+
The fast query consumes just a few fractions of a second, uses a join on varchar(32) columns and the joined table has 1352 records:
SELECT `inv_assets`.`id` AS `id`, `guarantor`.`lastname` AS `guarantoruname`,
(SELECT COALESCE(DATE_FORMAT(CONVERT_TZ(MIN(inspdate),'UTC','Europe/Vienna'),'%Y-%m-%d'),'')
FROM `mobuto_inv_inspections` AS `nextinsp`
LEFT JOIN `users` AS `saveuser`
ON (`saveuser`.`uid` = `nextinsp`.`saveuser`
AND `saveuser`.`_state` = '0')
WHERE ((`nextinsp`.`objectlink` = `inv_assets`.`id`
AND `nextinsp`.`inspdate` >= NOW()))
) AS `nextinsp`
FROM `mobuto_inv_assets` AS `inv_assets`
LEFT JOIN `users` AS `guarantor`
ON (`guarantor`.`uid` = `inv_assets`.`guarantor`
AND `guarantor`.`_state` = '0')
ORDER BY `inv_assets`.`type` ASC LIMIT 0, 20;
+----+--------------------+------------+--------+----------------+---------+---------+---------------------------------+-------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+--------+----------------+---------+---------+---------------------------------+-------+----------------+
| 1 | PRIMARY | inv_assets | ALL | NULL | NULL | NULL | NULL | 24857 | Using filesort |
| 1 | PRIMARY | guarantor | eq_ref | PRIMARY,_state | PRIMARY | 98 | mobuto_dev.inv_assets.guarantor | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | nextinsp | ALL | inspdate | NULL | NULL | NULL | 915 | Using where |
| 2 | DEPENDENT SUBQUERY | saveuser | eq_ref | PRIMARY,_state | PRIMARY | 98 | mobuto_dev.nextinsp.saveuser | 1 | Using where |
+----+--------------------+------------+--------+----------------+---------+---------+---------------------------------+-------+----------------+
The strange thing to me is, when I remove the column (description) of the joined table in the 'column-select-part' (while the join still persists and IMHO mysql does not optimize it away when not used), the speed is back (because mysql does not use a temporary table any longer and the explain looks same as the fast one, having type=eq_ref).
But why does this work for the first sample only when no column selected, whereas I can select one in the second one!?
CREATE TABLE `mobuto_inv_assets` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`invnum` varchar(10) NOT NULL,
`oebglcat` varchar(4) NOT NULL,
`mark` varchar(100) NOT NULL,
`type` varchar(100) NOT NULL,
`serialnum` varchar(100) NOT NULL,
`desc` varchar(100) NOT NULL,
`site` int(11) NOT NULL DEFAULT '0',
`licnum` varchar(20) NOT NULL DEFAULT '',
`inquirer` varchar(100) NOT NULL DEFAULT '',
`inqdate` date NOT NULL DEFAULT '0000-00-00',
`supplier` varchar(100) NOT NULL DEFAULT '',
`suppldate` date NOT NULL DEFAULT '0000-00-00',
`supplnumber` varchar(30) NOT NULL DEFAULT '',
`invoicedate` date NOT NULL DEFAULT '0000-00-00',
`invoicenumber` varchar(30) NOT NULL DEFAULT '',
`purchaseprice` decimal(11,2) NOT NULL DEFAULT '0.00',
`leased` varchar(1) NOT NULL DEFAULT 'N',
`leasingcompany` varchar(100) NOT NULL DEFAULT '',
`leasingnumber` varchar(30) NOT NULL DEFAULT '',
`notes` text NOT NULL,
`inspnotes` text NOT NULL,
`inactive` varchar(1) NOT NULL DEFAULT 'N',
`maintain` varchar(1) NOT NULL DEFAULT 'Y',
`asset` varchar(1) NOT NULL DEFAULT 'Y',
`inspection` varchar(1) NOT NULL DEFAULT '',
`inspperson` varchar(100) NOT NULL DEFAULT '',
`guarantor` varchar(32) NOT NULL DEFAULT '',
`saveuser` varchar(32) NOT NULL,
`savetime` int(11) NOT NULL,
`recordid` varchar(32) NOT NULL,
`_state` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `invnum` (`invnum`),
KEY `_state` (`_state`),
KEY `site` (`site`)
) ENGINE=InnoDB AUTO_INCREMENT=30707 DEFAULT CHARSET=utf8;
CREATE TABLE `mobuto_inv_sites` (
`siteid` int(11) NOT NULL AUTO_INCREMENT,
`description` varchar(100) NOT NULL,
`saveuser` varchar(32) NOT NULL,
`savetime` int(11) NOT NULL,
`recordid` varchar(32) NOT NULL,
`_state` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`siteid`),
KEY `_state` (`_state`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;
mysql> SHOW INDEX FROM mobuto_inv_assets;
+-------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mobuto_inv_assets | 0 | PRIMARY | 1 | id | A | 24857 | NULL | NULL | | BTREE | | |
| mobuto_inv_assets | 0 | invnum | 1 | invnum | A | 24857 | NULL | NULL | | BTREE | | |
| mobuto_inv_assets | 1 | _state | 1 | _state | A | 4 | NULL | NULL | | BTREE | | |
+-------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Changes as requested by #Wilson Hauck:
Added index to column site in mobuto_inv_assets (reduced execution speed by almost half a second)
Seems that column nextinsp was missing in first query. Maybe lost while formatting the query. Of course there should be the same as in the fast one
Removed the saveuser join as it is not used there (saved another 2 seconds) and updated its EXPLAIN (last line removed)
SHOW INDEX FROM mobuto_inv_sites added
+------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mobuto_inv_sites | 0 | PRIMARY | 1 | siteid | A | 3 | NULL | NULL | | BTREE | | |
| mobuto_inv_sites | 1 | _state | 1 | _state | A | 3 | NULL | NULL | | BTREE | | |
+------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Your first query is making less use of keys than the second. The possible_keys column in the explain plan shows where keys are available to be used, however, the key column shows where they are actually being used.
I would advise, short of seeing the structure of your DB, to make more use of these keys in your JOIN and WHERE clauses to speed it up.
I'd make sure that the query isn't being cached when you say you're modifying the select columns and the speed is varying.
12 seconds first query likely caused by ROWS column clues of simply 24857*3*915*1 = 68,232,465 total rows considered. Less than 1 second for second query ROWS column clues of simply 24857*1*915*1 = 22,744,155 total rows considered. The first query's use of Block Nested Loop processing is a major contributor to delaying the response.
Please post results of SHOW CREATE TABLE mobuto_inv_assets and mobuto_inv_sites. Please also post results of SHOW INDEX FROM mobuto_inv_assets and mobuto_inv_sites. With this additional information someone may be able to suggest improvements to SELECT .... queries that will avoid Block Nested Loop processing that is very time CPU intense with RBAR (Row By Agonizing Row processing). Additional indexing may be required.
Thanks for posting your two SHOW CREATE TABLE's, immensely helpful.
Please consider adding index with
ALTER TABLE mobuto_inv_sites ADD INDEX site --
if space permits on your system.
Also, the EXPLAIN showing for query1 is mismatched to the query.
The query does not refer to nextinsp or saveused that I can see in EXPLAIN.
Please replace the EXPLAIN for query1 after creating the index when you have an opportunity to test again and indicate any reduction in execution time required.
It would also be nice if you could post results of
SHOW INDEX FROM mobuto_inv_sites so we can see the scope of your data and cardinality.
If the inv_assets rows are populated with ACCURATE _state data
consider changing query1 to something like the following:
SELECT inv_assets.id AS id, site.description AS sitename,
(SELECT COALESCE(DATE_FORMAT(CONVERT_TZ(MIN(inspdate),'UTC','Europe/Vienna'),'%Y-%m-%d'),'')
FROM mobuto_inv_inspections AS nextinsp
WHERE ((nextinsp.objectlink = inv_assets.id
AND nextinsp.inspdate >= NOW()))
) AS nextinsp
FROM mobuto_inv_assets AS inv_assets
WHERE inv_assets._state = 2 OR inv_assets._state = 0
LEFT JOIN mobuto_inv_sites AS site
ON (site.siteid = inv_assets.site
AND site._state IN (2,0))
ORDER BY inv_assets.type ASC LIMIT 0, 20;
EXPLAIN should avoid table scan and subsequent Block Nested Loop processing.
If _state data in inv_assets is not ACCURATE on every row, this will not work.
2017-08-10 update 09:42 CT please post QUERY, EXPLAIN result, SHOW CREATE TABLE tblname for tables involved and SHOW INDEX FROM tblname for tables involved.
I have huge table with about 40 million rows (GPS tracker positions), recorded every 10 seconds from multiple devices inside company. I want to select only the first row of every minute, so I used group by. The problem is that the table is growing up every 10 seconds, I've tried almost everything, googled many hours. So I decided to ask a question.
I'm using MySQL 5.7.11 InnoDB pool 50GB, server is Xeon X5650 with 64GB RAM.
table structure:
CREATE TABLE `eventData` (
`id` bigint(20) NOT NULL,
`position` point NOT NULL,
`speed` decimal(6,2) DEFAULT NULL,
`time` datetime DEFAULT NULL,
`device_id` int(9) DEFAULT NULL,
`processed` tinyint(1) NOT NULL DEFAULT '0',
`time_m` datetime GENERATED ALWAYS AS ((`time` - interval second(`time`) second)) VIRTUAL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_czech_ci ROW_FORMAT=DYNAMIC;
ALTER TABLE `eventData`
ADD PRIMARY KEY (`id`),
ADD KEY `time` (`time`),
ADD KEY `device_id` (`device_id`,`processed`),
ADD KEY `time_m` (`time_m`);
SQL:
SELECT e.time, e.time_m, X(e.position) AS lat, Y(e.position) AS lng
FROM eventData AS e
WHERE
e.device_id = 86 AND
e.time BETWEEN '2016-02-29' AND '2016-03-06'
GROUP BY DAY(e.time),HOUR(e.time),MINUTE(e.time);
Explain:
EXPLAIN SELECT e.time, e.time_m, X(e.position) AS lat, Y(e.position) AS lng FROM eventData AS e WHERE e.device_id = 86 AND e.time BETWEEN '2016-02-29' AND '2016-03-06' GROUP BY DAY(e.time),HOUR(e.time),MINUTE(e.time);
+----+-------------+-------+------------+------+----------------+-----------+---------+-------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+----------------+-----------+---------+-------+---------+----------+---------------------------------------------------------------------+
| 1 | SIMPLE | e | NULL | ref | time,device_id | device_id | 5 | const | 2122632 | 6.40 | Using index condition; Using where; Using temporary; Using filesort |
+----+-------------+-------+------------+------+----------------+-----------+---------+-------+---------+----------+---------------------------------------------------------------------+
describe:
DESCRIBE eventData;
+------------------+------------------------+------+-----+---------+-------------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+------------------------+------+-----+---------+-------------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| position | point | NO | | NULL | |
| speed | decimal(6,2) | YES | | NULL | |
| time | datetime | YES | MUL | NULL | |
| device_id | int(9) | YES | MUL | NULL | |
| processed | tinyint(1) | NO | | 0 | |
| time_m | datetime | YES | MUL | NULL | VIRTUAL GENERATED |
+------------------+------------------------+------+-----+---------+-------------------+
I've tried:
without group by: ~0.06s
group by day,hour,minute: ~4.76s
group by virtual column (time_m): ~4.92s
group by e.time DIV 500: ~5.02s
I need to achieve better results than 5 seconds. Please help.
You could partition the table. For example by year. This would dramatically increase the performance due to much smaller indexes.
If this is not possible on your environment, try
GROUP BY date_format(e.time,'%d%H%i');
1) You can try composite index (device_id, time)
2) Try to group by virtual field:
SELECT MIN(e.time), e.time_m, X(e.position) AS lat, Y(e.position) AS lng
FROM eventData AS e
WHERE
e.device_id = 86 AND
e.time BETWEEN '2016-02-29' AND '2016-03-06'
GROUP BY e.time_m;
Here is my query which takes more than 5 second s to fetch 10 records n time gets bigger by changing the offsets in limit clause.
Table contain 12 million records.
SELECT device_id
,media_id
,limit1.play_date
,limit1.start_time
,limit1.end_time
,SUBTIME(limit1.end_time, limit1.start_time) AS playback_duration
FROM device_media_log
INNER JOIN (
SELECT play_date
,start_time
,end_time
,device_media_id
FROM device_media_log
ORDER BY play_date DESC
,start_time DESC
,end_time DESC limit 0
,10
) AS limit1 ON device_media_log.device_media_id = limit1.device_media_id;
explain plan::
+----+-------------+------------------+--------+---------------+---------+---------+------------------------+---------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+--------+---------------+---------+---------+------------------------+---------+----------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 10 | |
| 1 | PRIMARY | device_media_log | eq_ref | PRIMARY | PRIMARY | 8 | limit1.device_media_id | 1 | |
| 2 | DERIVED | device_media_log | ALL | NULL | NULL | NULL | NULL | 8345645 | Using filesort |
+----+-------------+------------------+--------+---------------+---------+---------+------------------------+---------+----------------+
here is create table::
CREATE TABLE `device_media_log` (
`device_media_id` bigint(20) NOT NULL AUTO_INCREMENT,
`device_id` int(11) NOT NULL DEFAULT '0',
`media_id` bigint(20) NOT NULL DEFAULT '0',
`playback_type_id` tinyint(4) NOT NULL DEFAULT '0',
`playback_id` int(11) NOT NULL DEFAULT '0',
`play_date` date DEFAULT NULL,
`start_time` time DEFAULT NULL,
`end_time` time DEFAULT NULL,
`client_id` bigint(20) DEFAULT NULL,
PRIMARY KEY (`device_media_id`),
KEY `Index_media_id` (`media_id`),
KEY `Index_device_id` (`device_id`),
KEY `Index_play_date` (`play_date`),
KEY `Index_start_time` (`start_time`),
KEY `Index_end_time` (`end_time`),
KEY `Index_client_id` (`client_id`)
)
ENGINE=InnoDB AUTO_INCREMENT=8366229 DEFAULT CHARSET=latin1
Describe after adding compound index
-+-------+---------------+----------------- +---------+------+------+-------+
| 1 | SIMPLE | device_media_log | index | NULL | index_composite | 12 | NULL | 10 | |
Try this query
SELECT device_id
,media_id
,limit1.play_date
,limit1.start_time
,limit1.end_time
,SUBTIME(limit1.end_time, limit1.start_time) AS playback_duration
FROM
device_media_log
ORDER BY
play_date DESC
,start_time DESC
,end_time DESC
limit 0, 10;
There is no need of subquery as you are directly using the result.
Also the explain statement shows that none of your created index is used.
Create a compound index on following column play_date, start_time, end_time.
ALTER TABLE device_media_log ADD INDEX device_media_log_date_time(play_date, start_time, end_time);
hope this helps...