I have two columns in my MySQL Database start_time and end_time. They both are TIMESTAMP. I have created an index on the two columns.
I have 3.5 million rows. This query takes 13s to be executed :
select * from test WHERE start_time > TIMESTAMP('2020-04-02 09:00:00') and end_time < TIMESTAMP('2020-04-02 10:00:00')
Is there any way to optimise it ?
EDIT:
CREATE TABLE `test` (
`YYY` varchar(255) NOT NULL,
`start_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
UNIQUE KEY `index1` (`YYY`,`start_time`) USING BTREE,
UNIQUE KEY `index2` (`YYY`,`start_time`,`end_time`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC
Don't bother having one UNIQUE (index2) that starts with the same column as another UNIQUE (index1).
There is no index (in MySQL, at least) that optimizes for such an "overlap" test.
If you the time ranges are non-overlapping (that is, no two pairs of start..end overlap other than one end matches another `start), then my technique for IP-ranges scales quite well. But it requires restructuring the table. http://mysql.rjweb.org/doc.php/ipranges
I cannot replicate this finding:
EXPLAIN
-> SELECT *
-> FROM test
-> WHERE start_time >= '2020-04-02 09:00:00'
-> AND end_time <= '2020-04-02 10:00:00';
+----+-------------+-------+-------+---------------+--------+---------+------+--------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+--------+---------+------+--------+--------------------------+
| 1 | SIMPLE | test | index | NULL | index2 | 267 | NULL | 262035 | Using where; Using index |
+----+-------------+-------+-------+---------------+--------+---------+------+--------+--------------------------+
1 row in set (0.00 sec)
Actually, I'm surprised that the index is used, given that YYY is not part of the filtering condition. Anyway, try without YYY, or try with YYY as the third argument in the index. And remove the other indices.
Related
Version - 10.4.25-MariaDB
I have a table where column(name) is a second part of primary key(idarchive,name).
When i run count(*) on table where name like 'done%', its using the index on field name properly but when i run select * its not using the separate index instead using primary key and slowing down the query.
Any idea what we can do here ?
any changes in optimizer switch or any other alternative which can help ?
Note - we can't use force index as queries are not controlable.
Table structure:
CREATE TABLE `table1 ` (
`idarchive` int(10) unsigned NOT NULL,
`name` varchar(255) NOT NULL,
`idsite` int(10) unsigned DEFAULT NULL,
`date1` date DEFAULT NULL,
`date2` date DEFAULT NULL,
`period` tinyint(3) unsigned DEFAULT NULL,
`ts_archived` datetime DEFAULT NULL,
`value` double DEFAULT NULL,
PRIMARY KEY (`idarchive`,`name`),
KEY `index_idsite_dates_period` (`idsite`,`date1`,`date2`,`period`,`ts_archived`),
KEY `index_period_archived` (`period`,`ts_archived`),
KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Queries:
explain select count(*) from table1 WHERE name like 'done%' ;
+------+-------------+-------------------------------+-------+---------------+------+---------+------+---------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+-------+---------------+------+---------+------+---------+--------------------------+
| 1 | SIMPLE | table1 | range | name | name | 767 | NULL | 9131455 | Using where; Using index |
+------+-------------+-------------------------------+-------+---------------+------+---------+------+---------+--------------------------+
1 row in set (0.000 sec)
explain select * from table1 WHERE name like 'done%' ;
+------+-------------+-------------------------------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+------+---------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | table1 | ALL | name | NULL | NULL | NULL | 18262910 | Using where |
+------+-------------+-------------------------------+------+---------------+------+---------+------+----------+-------------+
1 row in set (0.000 sec) ```
Your SELECT COUNT(*) ... LIKE 'constant%' query is covered by your index on your name column. That is, the entire query can be satisfied by reading the index. So the query planner decides to range-scan your index to generate the result.
On the other hand, your SELECT * query needs all columns from all rows of the table. That can't be satisfied from any of your indexes. And, it's possible your WHERE name like 'done%' filter reads a significant fraction of the table, enough so the query planner decides the fastest way to satisfy it is to scan the entire table. The query planner figures this out by using statistics on the contents of the table, plus some knowledge of the relative costs of IO and CPU.
If you have just inserted many rows into the table you could try doing ANALYZE TABLE table1 and then rerun the query. Maybe after the table's statistics are updated you'll get a different query plan.
And, if you don't need all the columns, you could stop using SELECT * and instead name the columns you need. SELECT * is a notorious query-performance antipattern, because it often returns column data that never gets used. Once you know exactly what columns you want, you could create a covering index to provide them.
These days the query planner does a pretty good job of optimizing simple queries such as yours.
In MariaDB you can say ANALYZE FORMAT=JSON SELECT.... It will run the query and show you details of the actual execution plan it used.
Server version:
[root#cat best]# /usr/libexec/mysqld --version
/usr/libexec/mysqld Ver 5.1.47 for redhat-linux-gnu on i386 (Source distribution)
Schema:
CREATE TABLE `Log` (
`EntryId` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`EntryTime` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP(),
`Severity` ENUM(
'LOG_LEVEL_CRITICAL',
'LOG_LEVEL_ERROR',
'LOG_LEVEL_WARNING',
'LOG_LEVEL_NOTICE',
'LOG_LEVEL_INFO',
'LOG_LEVEL_DEBUG'
) NOT NULL,
`User` TEXT,
`Text` TEXT NOT NULL,
PRIMARY KEY(`EntryId`),
KEY `TimeId` (`EntryTime`,`EntryId`)
) ENGINE=InnoDB COMMENT="Log of server activity";
Query:
SELECT
`EntryId`,
`EntryTime`, -- or, ideally: UNIX_TIMESTAMP(`EntryTime`) AS `EntryTime_UnixTS`
`Severity`,
`User`,
`Text`
FROM `Log`
ORDER BY `EntryTime` DESC, `EntryId` DESC
LIMIT 0, 20
According to the execution plan (and observation), the index is not being used:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE Log ALL \N \N \N \N 720 Using filesort
I've tried re-organising it a few ways with little success but, more than anything, would like to understand why this simple approach is failing. My understanding was that a left-most subset of any key can be used to optimise an ORDER BY operation.
Is my index wrong? Can I optimise the query?
Please note that I will also want to conditionally add, e.g.
WHERE `Severity` <= 'LOG_LEVEL_WARNING'
though I'd like to get the basic version working first if this makes the solution very different.
Reproduced on SQLFiddle under MySQL 5.5.32.
The reason is that you index includes the primary key in it. and since it is InnoDB, by default the PK is ancluded in all other indexes as the left-most field. i.e. the index in this case is (EntryId, EntryTime, EntryId).
The solution is to have this index only on (EntryTime):
alter table Log drop index TimeId;
alter table Log add index TimeId(EntryTime);
explain SELECT `EntryId`, `EntryTime`, `Severity`, `User`, `Text` FROM `Log` ORDER BY `EntryTime` DESC, `EntryId` DESC LIMIT 0, 20;
+----+-------------+-------+-------+---------------+--------+---------+------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+--------+---------+------+------+-------+
| 1 | SIMPLE | Log | index | NULL | TimeId | 4 | NULL | 20 | NULL |
+----+-------------+-------+-------+---------------+--------+---------+------+------+-------+
HTH
my query is running longer than 30 minutes .it is a simple query even it contains indexes also.we are unable to find why it was taking too much execution time and it effects on our entire db performance.
yesterday it ran around: 122.6mins
any one can help me here.how to improve query performance
This is my query:
SELECT tab1.customer_id,tab1.row_mod,tab1.row_create,tab1.event_id,tab1.event_type,
tab1.new_value,tab1.old_value FROM tab1 force index (tab1_n2)where customer_id >= 1 and customer_id
< 5000000 and (tab1.row_mod >= '2012-10-01') or (tab1.row_create >= '2012-10-01' and tab1.row_create < '2012-10-13');
Explain plan
+----+-------------+------------------+------+---------------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+---------------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | tab1 | ALL | tab1_n2 | NULL | NULL | NULL | 18490530 | Using where |
+----+-------------+------------------+------+---------------------+------+---------+------+----------+-------------+
1 row in set (0.00 sec)
Table structure:
mysql> show create table tab1\G
*************************** 1. row ***************************
Table: tab1
Create Table: CREATE TABLE `tab1` (
`customer_id` int(11) NOT NULL,
`row_mod` datetime DEFAULT NULL,
`row_create` datetime DEFAULT NULL,
`event_id` int(11) DEFAULT NULL,
`event_type` varchar(45) DEFAULT NULL,
`new_value` varchar(255) DEFAULT NULL,
`old_value` varchar(255) DEFAULT NULL,
KEY `customer_id1` (`customer_id`),
KEY `new_value_n1` (`new_value`),
KEY `tab1_n1` (`row_create`),
KEY `tab1_n2` (`row_mod`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
Please help me how to tune it .even it having indexes also
Probably because you are using an index that does not make sense.
The row_mod condition is only one branch of the OR condition, so that index is not much help here. If you are forcing every lookup through the index without eliminating any rows, that could be a lot slower than a full table scan. Good rule of thumb is that an index should eliminate more than 90% of rows.
Try to do without the "force index" part.
Try using a UNION of the two conditions. That way each condition can use an index.
ALTER TABLE tab1 ADD INDEX idx_row_mod_customer_id (row_mod, customer_id);
ALTER TABLE tab1 ADD INDEX idx_row_create (row_create);
SELECT tab1.customer_id, tab1.row_mod, tab1.row_create, tab1.event_id, tab1.event_type,
tab1.new_value, tab1.old_value
FROM tab1
WHERE customer_id >= 1 and customer_id
< 5000000 AND tab1.row_mod >= '2012-10-01'
UNION
SELECT tab1.customer_id, tab1.row_mod, tab1.row_create, tab1.event_id, tab1.event_type,
tab1.new_value, tab1.old_value
FROM tab1
WHERE tab1.row_create >= '2012-10-01' AND tab1.row_create < '2012-10-13';
To optimise further, you could add all selected columns to both indices, saving MySQL from having to load the rows into memory. This will greatly increase the size of the indices, and therefore their memory requirement.
I'm hoping (and pretty sure that) someone out there is much better at MySQL queries than msyelf.
I have a query which checks a table that contains information on :
- a search term
- title and price results from various sites using this search term
For the sake of streamlining, I've inserted the data already converted to lowercase with spaces removed and the whole thing trimmed to 11 characters to help reduce the load on the MySQL server.
The query is designed to find the maximum cost and minimum cost of likely equal titles and determine a price difference if it exists.
Having read some similar questions here, I've also prepended EXPLAIN EXTENDED to the query to see if that would help and I'm including the results along with the query.
The query as is :
SELECT
a.pricesrch11,
b.pricesrch11,
a.pricegroup11,
b.pricegroup11,
a.priceamt - b.priceamt AS pricediff
FROM ebssavings a
LEFT JOIN ebssavings b ON ( a.pricesrch11 = b.pricesrch11 )
AND (a.pricegroup11 = a.pricesrch11)
AND (b.pricegroup11 = a.pricesrch11)
WHERE a.priceamt - b.priceamt >0
GROUP BY a.pricesrch11
The results of the EXPLAIN :
select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
1 | SIMPLE | a | ALL | pricesrch11,pricegroup11 | NULL | NULL | NULL | 8816 | Using where; Using temporary; Using filesort
1 | SIMPLE | b | ALL | pricesrch11,pricegroup11 | NULL | NULL | NULL | 6612 | Using where
ADDENDUM :
I just ran this query and got the following result :
Showing rows 0 - 4 ( 5 total, Query took 66.8119 sec)
CREATE TABLE IF NOT EXISTS ebssavings
( priceid int(44) NOT NULL auto_increment,
priceamt decimal(10,2) NOT NULL,
pricesrch11 varchar(11) character set utf8 collate utf8_unicode_ci NOT NULL,
pricegroup11 varchar(11) character set utf8 collate utf8_unicode_ci NOT NULL,
pricedate timestamp NOT NULL default CURRENT_TIMESTAMP,
PRIMARY KEY (priceid),
KEY priceamt (priceamt),
KEY pricesrch11 (pricesrch11),
KEY pricegroup11 (pricegroup11) )
ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=8817
MORE INFO ON THE NEW INDEXES (removed pricegroup11, and made a composite index called srchandtitle from pricesrch11 and pricegroup11):
Edit Drop PRIMARY BTREE Yes No priceid 169 A
Edit Drop priceamt BTREE No No priceamt 56 A
Edit Drop pricesrch11 BTREE No No pricesrch11 12 A
Edit Drop srchandtitle BTREE No No pricesrch11 12 A
pricegroup11 169 A
Create two indexes:
PriceSrch11
A clustered index on pricesrch11,pricegroup11
Remove the Key on pricegroup11 and add a composite clustered key on pricesrch11,pricegroup11.
Also move the table to InnoDB.
It seems that things have sped up now with the changes made to the table and the indexes.
I've emptied the table and am beginning again.
Thank you all for your help.
-A
Probably I'm missing some silly thing... Apparently MySQL 5.1 keeps doing a Filesort even when there is an index that matches exactly the column in the ORDER BY clause. To post it here, I've oversimplified the data model, but the issue is still happening:
Table definition:
CREATE TABLE `event` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`owner_id` int(11) DEFAULT NULL,
`date_created` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `owner_id` (`owner_id`),
KEY `date_created` (`date_created`),
CONSTRAINT `event_ibfk_1` FOREIGN KEY (`owner_id`) REFERENCES `user_profile` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8;
My problem is that event a simple SELECT is showing "Using filesort":
explain select * from event order by date_created desc;
And the result for the query explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE event ALL NULL NULL NULL NULL 6 Using filesort
Is there any way for this type of queries to use the index insteas of doing a filesort?
Thanks in advance to everybody.
Since your CREATE TABLE statement indicates that you have less than 10 rows (AUTO_INCREMENT=7) and using FORCE INDEX on my installation will make MySQL use the index, I'm guessing the optimizer thinks a table scan is faster (less random I/O) than an index scan (since you're selecting all columns, not just date_created). This is confirmed by the following:
mysql> explain select date_created from event order by date_created;
+----+-------------+-------+-------+---------------+--------------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+--------------+---------+------+------+-------------+
| 1 | SIMPLE | event | index | NULL | date_created | 9 | NULL | 1 | Using index |
+----+-------------+-------+-------+---------------+--------------+---------+------+------+-------------+
1 row in set (0.00 sec)
In the above case, the index scan is faster because only the indexed column needs to be returned.
The MySQL documentation has some cases where using an index is considered slower: http://dev.mysql.com/doc/refman/5.1/en/how-to-avoid-table-scan.html
Q: Is there any way for this type of queries to use the index instead of doing a filesort?
A: To have MySQL use the index if at all possible, try:
EXPLAIN SELECT * FROM event FORCE INDEX (date_created) ORDER BY date_created DESC;
By using the FORCE INDEX (index_name), this tells MySQL to make use of the index if it's at all possible. Absent that directive, MySQL will choose the most efficient way to return the result set. A filesort may be more efficient than using the index.