I have a table
CREATE TABLE tb_incident (
id int(10) unsigned NOT NULL AUTO_INCREMENT,
what tinyint(4) NOT NULL,
when bigint(20) unsigned NOT NULL,
where tinytext NOT NULL,
where_longitude_latitude point NOT NULL,
who tinyint(4) DEFAULT NULL,
why tinyint(4) DEFAULT NULL,
description text,
created timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (id),
UNIQUE KEY id_UNIQUE (id),
KEY index_main_items (what,when),
SPATIAL KEY location_index (where_longitude_latitude)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
When I try running this query,
SELECT db_incident.distance(point(132.6,15.604436765987833),where_longitude_latitude,'mi') AS "Distance in miles",
X(where_longitude_latitude) AS "Longitude",
Y(where_longitude_latitude) AS "Latitude"
FROM db_incident.tb_incident
WHERE db_incident.distance(point(132.6,15.604436765987833),where_longitude_latitude,'mi') <= 5
LIMIT 100
It takes over 10 minutes to execute, sometimes it gives this error Error Code: 2013. Lost connection to MySQL server during query
My table has over 10 million records. Can someone please take a look and see if there is a way the query can be optimized.
NOTE: db_incident.distance is a function that returns the distance between two points.
Below is the EXPLAIN of the query.
+----+-------------+-------------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+------+---------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | tb_incident | ALL | NULL | NULL | NULL | NULL | 11689629 | Using where |
+----+-------------+-------------+------+---------------+------+---------+------+----------+-------------+
1 row in set (0.00 sec)
Related
Mysql version - 5.7.22
Table definition
CREATE TABLE `books` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`uuid` char(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`title` varchar(254) COLLATE utf8mb4_unicode_ci NOT NULL,
`created` datetime(6) NOT NULL,
`modified` datetime(6) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `book_uuid` (`uuid`),
) ENGINE=InnoDB AUTO_INCREMENT=115 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Query
SELECT DISTINCT `books`.`id`,
`books`.`uuid`,
`books`.`title`
FROM `books`
WHERE (`books`.`uuid` IN ("334222a0e99b4a3e97f577665055208e",
"979c059840964934816280ba85c67221",
"4e2978c765dd435998666ea3083666e5",
"535aa78ba80e4215bbf75fb1e20cc5f3",
"f969fb10c72b4875aabdf75c1b493524",
"1daa0015055444a4b1c0821618a7a4d9",
"04f34ede284a4b86b0adddb405d30a75",
"513cad12c88c44c6ab248d43643459b9",
"de2bde6d016f4381ad0ba714234386fa",
"f645c2c9f1594a199a960b97b7015986",
"3ce02c072f24447a8a7b269a19ec554f",
"75450daf9d024d9d9c0df038437ae2c2",
"0e822042b50b4f79bb38304e0acde6f0",
"38d808fb3f9a4f57b4f7b30a141e7169",
"ecd424abd3a94a339383f6f8e668655e"))
ORDER BY `books`.`id` DESC
LIMIT 15;
when i do explain on this query it doesn't pick the index
+----+-------------+-----------------+------------+------+---------------+------+---------+------+------+----------+-----------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------+------------+------+---------------+------+---------+------+------+----------+-----------------------------+
| 1 | SIMPLE | books | NULL | ALL | book_uuid | NULL | NULL | NULL | 107 | 12.15 | Using where; Using filesort |
+----+-------------+-----------------+------------+------+---------------+------+---------+------+------+----------+-----------------------------+
strangely the index is correctly used when there are 12 or less entries passed to the IN clause (without forcing index)
forcing the index works, but I cannot force index as this query is created by django ORM, cannot use django-mysql 's force_index as I am on Innodb
Note: I know 'distinct' and 'limit' can be avoided in this query, but its part of a bigger query, so i have kept it as is
I was migrating from my old MariaDB 10.0 database to new Google Cloud Sql with Mysql 5.7 with mysqldump method. After migrating, i got some very slow query regarding with WHERE EXISTS statement.
I tried to EXPLAIN my query on both my old DB and new DB and it explained different result. Since it using dump, i am assuming that no changes with the table indexes. This is the query that i wanted to run
SELECT * FROM detitem
where exists (select 1 from detlayanan
where detitem.iddetlayanan = detlayanan.id
and detlayanan.layanan_idlayanan='LYN15176176101503')
the EXPLAIN from old DB
+------+-------------+------------+------+------------------------------------+----------------------------+---------+--------------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+------------+------+------------------------------------+----------------------------+---------+--------------------------------+-------+--------------------------+
| 1 | PRIMARY | detlayanan | ref | PRIMARY,fk_detlayanan_layanan1_idx | fk_detlayanan_layanan1_idx | 22 | const | 11030 | Using where; Using index |
| 1 | PRIMARY | detitem | ref | FK_detitem_detlayanan | FK_detitem_detlayanan | 52 | citridia_sinadme.detlayanan.id | 1 | |
+------+-------------+------------+------+------------------------------------+----------------------------+---------+--------------------------------+-------+--------------------------+
and the EXPLAIN from new DB
+----+--------------------+------------+------------+--------+------------------------------------+---------+---------+---------------------------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+------------+------------+--------+------------------------------------+---------+---------+---------------------------------------+---------+----------+-------------+
| 1 | PRIMARY | detitem | NULL | ALL | NULL | NULL | NULL | NULL | 2079094 | 100.00 | Using where |
| 2 | DEPENDENT SUBQUERY | detlayanan | NULL | eq_ref | PRIMARY,fk_detlayanan_layanan1_idx | PRIMARY | 52 | citridia_sinadme.detitem.iddetlayanan | 1 | 5.00 | Using where |
+----+--------------------+------------+------------+--------+------------------------------------+---------+---------+---------------------------------------+---------+----------+-------------+
The new one is doing Full-table scan even there is index exist. Am i missing something here?
Here is the "detlayanan" table
CREATE TABLE `detlayanan` (
`transaksi_idtransaksi` varchar(40) NOT NULL,
`layanan_idlayanan` varchar(20) NOT NULL,
`nama_layanan` varchar(255) DEFAULT NULL,
`jumlah_beli` float DEFAULT NULL,
`id` varchar(50) NOT NULL,
`harga` decimal(20,2) DEFAULT '0.00',
`hargatotal` decimal(20,2) DEFAULT '0.00',
`luas_p` double(255,2) DEFAULT '0.00',
`luas_l` double(255,2) DEFAULT '0.00',
`luas_q` double(255,2) DEFAULT '0.00',
`keterangan` varchar(255) DEFAULT '',
`iddeposit` varchar(255) DEFAULT NULL,
`posisi` tinyint(4) DEFAULT '1',
`idworkshop` varchar(60) DEFAULT NULL,
`is_wsot` tinyint(4) DEFAULT '0',
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP,
`total_bersih` varchar(20) DEFAULT '0',
`total_min_order` decimal(20,2) DEFAULT '0.00',
`kondisi_barang` text,
PRIMARY KEY (`id`),
KEY `fk_detlayanan_layanan1_idx` (`layanan_idlayanan`),
KEY `fk_detlayanan_deposit` (`iddeposit`),
KEY `transaksi_idtransaksi` (`transaksi_idtransaksi`),
CONSTRAINT `detlayanan_ibfk_1` FOREIGN KEY (`transaksi_idtransaksi`) REFERENCES `transaksi` (`idtransaksi`),
CONSTRAINT `fk_detlayanan_layanan1` FOREIGN KEY (`layanan_idlayanan`) REFERENCES `layanan` (`idlayanan`) ON DELETE NO ACTION ON UPDATE NO ACTION
)
And here is the "detitem" table
CREATE TABLE `detitem` (
`item_iditem` varchar(20) NOT NULL,
`layanan_idlayanan` varchar(255) NOT NULL,
`jumlah_item` int(255) DEFAULT NULL,
`transaksi_idtransaksi` varchar(255) NOT NULL,
`id` varchar(50) NOT NULL,
`iddetlayanan` varchar(50) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP,
`hapus` tinyint(4) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `FK_detitem_item` (`item_iditem`),
KEY `FK_detitem_layanan` (`layanan_idlayanan`),
KEY `FK_detitem_transaksi` (`transaksi_idtransaksi`),
KEY `FK_detitem_detlayanan` (`iddetlayanan`),
CONSTRAINT `FK_detitem_detlayanan` FOREIGN KEY (`iddetlayanan`) REFERENCES `detlayanan` (`id`),
CONSTRAINT `FK_detitem_item` FOREIGN KEY (`item_iditem`) REFERENCES `item` (`iditem`),
CONSTRAINT `FK_detitem_layanan` FOREIGN KEY (`layanan_idlayanan`) REFERENCES `layanan` (`idlayanan`),
CONSTRAINT `FK_detitem_transaksi` FOREIGN KEY (`transaksi_idtransaksi`) REFERENCES `transaksi` (`idtransaksi`)
)
I expect the "rows" on explain stay small, in the old DB, "rows" stays at very low even in another table. but in the new DB it can shows up to million.
UPDATE
After some research, actually i must explicitly add some indexed column in the where statement to accompany EXISTS statement. so the query would be like this
SELECT * FROM detitem WHERE
<indexed column> in (<some id's>)
AND EXISTS ( SELECT 1 FROM detlayanan WHERE detitem.iddetlayanan =
detlayanan.id AND detlayanan.layanan_idlayanan = 'LYN15176176101503' )
apparently MySQL doing full scan on detitem table to check the subquery values is exists or not, when some indexed column are declared, sql does not need to did that. Also this case were found in MySQL.
some references:
https://mariadb.com/kb/en/library/exists-to-in-optimization/
This looks like a case where MariaDB's Optimizer is a step or two ahead of MySQL's.
See if this works well:
SELECT i.*
FROM ( SELECT id
FROM detlayanan
WHERE layanan_idlayanan = 'LYN15176176101503'
) AS x
JOIN detitem AS i ON x.id = i.iddetlayanan
I think it will work well in both servers.
Simpler yet:
SELECT i.*
FROM detlayanan AS lay
JOIN detitem AS i ON lay.id = i.iddetlayanan
WHERE lay.layanan_idlayanan = 'LYN15176176101503'
Please refer the table strcuture below.
CREATE TABLE `oarc` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`zID` int(11) NOT NULL,
`cID` int(11) NOT NULL,
`bID` int(11) NOT NULL,
`rtype` char(1) COLLATE utf8_unicode_ci NOT NULL,
`created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1821039 ;
Other than the PRIMARY KEY, I have not set any index on this, and when I run the following query
select COUNT(oarc.ID) as total
from `oarc` where`oarc`.`rtype` = 'v'
group
by `oarc`.`zID`
I am getting the result in less than 1 second. But if I add index to zID it is taking more than 5 seconds.
Please see below explain result :
id | select_type | table | type | possible_keys | key | key_len | ref | row | Extra
--------------------------------------------------------------------------------------------------------
1 | SIMPLE | oarc | index | NULL | zone_ID | 4 | NULL | 1909387 | Using where
Currently the table have more than 1821039 records in it and it will increase on a hourly basis. What are the things I need to do in order to reduce the query execution time. I am expecting only something at the table and query level, nothing on my.cnf or server side because I can not do anything there.
Thanks in advance.
Is this better?
CREATE TABLE `oarc` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`zID` int(11) NOT NULL,
`cID` int(11) NOT NULL,
`bID` int(11) NOT NULL,
`rtype` char(1) COLLATE utf8_unicode_ci NOT NULL,
`created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`ID`),
KEY(rtype,zid)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1821039 ;
explain
select COUNT(oarc.ID) as total
from `oarc` where`oarc`.`rtype` = 'v'
group
by `oarc`.`zID`
+----+-------------+-------+------+---------------+-------+---------+-------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+-------+---------+-------+------+--------------------------+
| 1 | SIMPLE | oarc | ref | rtype | rtype | 3 | const | 1 | Using where; Using index |
+----+-------------+-------+------+---------------+-------+---------+-------+------+--------------------------+
i have one MySQL issue. I have to optimize some queries on my website. One of them i have already done, but there are still some which i cannot resolve without your help.
I have a table called "news":
CREATE TABLE IF NOT EXISTS `news` (
`id` int(10) NOT NULL auto_increment,
`edited` smallint(1) NOT NULL default '0',
`site` varchar(30) default NULL,
`foreign_id` varchar(25) default NULL,
`title` varchar(255) NOT NULL,
`text` text NOT NULL,
`image` varchar(255) default NULL,
`horizontal` smallint(1) NOT NULL,
`image_author` varchar(255) default NULL,
`text_author` varchar(255) default NULL,
`lang` varchar(3) NOT NULL,
`link` varchar(255) NOT NULL,
`date` date NOT NULL,
`redirect` smallint(1) NOT NULL,
`parent` int(10) NOT NULL,
`views` int(5) NOT NULL,
`status` smallint(1) NOT NULL,
PRIMARY KEY (`id`),
KEY `lang` (`lang`,`status`),
KEY `date` (`date`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=47122 ;
as you can see i have two indexes: "lang" and "date"
I have tried some combinations of different indexes and this one has produced me the best results ... unfortunately only on my local computer. On the server i still have bad results. I want to say that the database is the same.
query:
SELECT id FROM news WHERE lang = 'en' AND STATUS =1 ORDER BY DATE DESC LIMIT 0, 10
localhost explain:
+----+-------------+-------+-------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | news | index | lang | date | 3 | NULL | 23 | Using where |
+----+-------------+-------+-------+---------------+------+---------+------+------+-------------+
server explain:
+----+-------------+-------+------+---------------+--------+---------+-------------+-------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+--------+---------+-------------+-------+-----------------------------+
| 1 | SIMPLE | news | ref | status | status | 13 | const,const | 15840 | Using where; Using filesort |
+----+-------------+-------+------+---------------+--------+---------+-------------+-------+-----------------------------+
I have looked a lot of other similar topics, but unfortunately i cannot find any solution to work on my server. I will be very glad to here from you some solution with some explanation for that so i can optimize my other queries.
Thanks !
This is your query:
SELECT id
FROM news
WHERE lang = 'en' AND STATUS =1
ORDER BY DATE DESC
LIMIT 0, 10
The best index is one that contains all the fields used in the query (four fields in all). The ordering in the index is by equality conditions in the where clause followed by the order by clause followed by other columns in the select clause.
So, try this index: ndws(leng, status, date, id).
Here is the query:
select timespans.id as timespan_id, count(*) as num
from reports, timespans
where timespans.after_date >= '2011-04-13 22:08:38' and
timespans.after_date <= reports.authored_at and
reports.authored_at < timespans.before_date
group by timespans.id;
Here are the table defs:
CREATE TABLE `reports` (
`id` int(11) NOT NULL auto_increment,
`source_id` int(11) default NULL,
`url` varchar(255) default NULL,
`lat` decimal(20,15) default NULL,
`lng` decimal(20,15) default NULL,
`content` text,
`notes` text,
`authored_at` datetime default NULL,
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
`data` text,
`title` varchar(255) default NULL,
`author_id` int(11) default NULL,
`orig_id` varchar(255) default NULL,
PRIMARY KEY (`id`),
KEY `index_reports_on_title` (`title`),
KEY `index_content_on_reports` (`content`(128))
CREATE TABLE `timespans` (
`id` int(11) NOT NULL auto_increment,
`after_date` datetime default NULL,
`before_date` datetime default NULL,
`after_offset` int(11) default NULL,
`before_offset` int(11) default NULL,
`is_common` tinyint(1) default NULL,
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
`is_search_chunk` tinyint(1) default NULL,
`is_day` tinyint(1) default NULL,
PRIMARY KEY (`id`),
KEY `index_timespans_on_after_date` (`after_date`),
KEY `index_timespans_on_before_date` (`before_date`)
And here is the explain:
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+----------------------------------------------+
| 1 | SIMPLE | timespans | range | index_timespans_on_after_date,index_timespans_on_before_date | index_timespans_on_after_date | 9 | NULL | 84 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | reports | ALL | NULL | NULL | NULL | NULL | 183297 | Using where |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+----------------------------------------------+
And here is the explain after I create an index on authored_at. As you can see, the index is not actually getting used (I think...)
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+------------------------------------------------+
| 1 | SIMPLE | timespans | range | index_timespans_on_after_date,index_timespans_on_before_date | index_timespans_on_after_date | 9 | NULL | 86 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | reports | ALL | index_reports_on_authored_at | NULL | NULL | NULL | 183317 | Range checked for each record (index map: 0x8) |
+----+-------------+-----------+-------+--------------------------------------------------------------+-------------------------------+---------+------+--------+------------------------------------------------+
There are about 142k rows in the reports table, and far fewer in the timespans table.
The query is taking about 3 seconds now.
The strange thing is that if I add an index on reports.authored_at, it actually makes the query far slower, about 20 seconds. I would have thought it would do the opposite, since it would make it easy to find the reports at either end of the range, and throw the rest away, rather than having to examine all of them.
Can someone clarify? I'm stumped.
Instead of two separate indexes for the timespan table, try merging them into a single multi-column index with before_date and after_date in a single index. Then add that index to authored_at as well.
i rewrite you query like this:
select t.id, count(*) as num from timespans t
join reports r where t.after_date >= '2011-04-13 22:08:38'
and r.authored_at >= '2011-04-13 22:08:38'
and r.authored_at < t.before_date
group by t.id order by null;
and change indexes of tables
alter table reports add index authored_at_idx(authored_at);
You can used partition feature of database on column after_date. It will help u a lot.