how to speed up the mysql query to count average value? - mysql

the table structure is follow:
CREATE TABLE `crm_member` (
`member_id` int(11) NOT NULL AUTO_INCREMENT,
`shop_id` int(11) NOT NULL,
`nick` varchar(255) NOT NULL DEFAULT '',
`name` varchar(255) NOT NULL DEFAULT '',
`mobile` varchar(255) NOT NULL DEFAULT '',
`grade` int(11) NOT NULL DEFAULT '-1',
`trade_count` int(11) NOT NULL,
`trade_amount` float NOT NULL,
`last_trade_time` int(11) NOT NULL,
`trade_from` tinyint(4) NOT NULL,
`avg_price` float NOT NULL,
`seller_flag` tinyint(1) NOT NULL,
`is_black` tinyint(1) NOT NULL DEFAULT '0',
`created` int(11) NOT NULL,
PRIMARY KEY (`member_id`),
UNIQUE KEY `shop_id` (`shop_id`,`nick`),
KEY `last_trade_time` (`last_trade_time`),
KEY `idx_shop_id_grade` (`shop_id`,`grade`),
KEY `idx_shopid_created` (`shop_id`,`created`),
KEY `idx_trade_amount` (`shop_id`,`trade_amount`),
KEY `idx_trade_count` (`shop_id`,`trade_count`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
the table has 2148037 rows under the shop_id 3498706;
AND my query is like
SELECT AVG(trade_count) as trade_count, AVG(trade_amount) as trade_amount, AVG(grade) as grade from `crm_member_0141` WHERE shop_id = '3498706' and grade >= 0 and trade_count > 0 and is_black = 0 LIMIT 1
the query exec about 30 seconds .
the explain result
mysql> explain SELECT member_id, AVG(trade_count) as trade_count, AVG(trade_amount) as trade_amount, AVG(grade) as grade from `crm_member_0141` WHERE shop_id = 3498706 and grade >= 0 and trade_count > 0 and is_black = 0 order by member_id LIMIT 1;
+----+-------------+-----------------+------------+------+-------------------------------------------------------------------------------+---------+---------+-------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------+------------+------+-------------------------------------------------------------------------------+---------+---------+-------+---------+----------+-------------+
| 1 | SIMPLE | crm_member_0141 | NULL | ref | shop_id,idx_shop_id_grade,idx_shopid_created,idx_trade_amount,idx_trade_count | shop_id | 4 | const | 1074018 | 1.11 | Using where |
+----+-------------+-----------------+------------+------+-------------------------------------------------------------------------------+---------+---------+-------+---------+----------+-------------+
how to speed up the query?

It looks like mysql chooses to use the primary key instead of the shop_id, grade index, although shop_id, grade would probably speed up the selection of rows.
You can tell mysql to use a specific index for a query, using the USE INDEX directive :
try to add USE INDEX (idx_shop_id_grade) in your query after the table name and see if the computation is faster.
Otherwise, if this query is especially useful and frequent to call in your app, you can build a more specialised index for this query :
Your query selects on a particular value for shop_id and is_black, then does a range selection on grade and trade_count.
I would suggest to try indexing using shop_id, is_black, grade, trade_count.
note : obviously, test this first on test database

Related

Slow MySQL order by datetime column

I have news table defined like this:
CREATE TABLE `news` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`creation_date` datetime DEFAULT NULL,
`modification_date` datetime DEFAULT NULL,
`active` bit(1) DEFAULT NULL,
`mark_for_delete` bit(1) DEFAULT NULL,
`verified` bit(1) DEFAULT NULL,
`bot_id` int(11) DEFAULT NULL,
`description` varchar(1000) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`hash` varchar(100) NOT NULL,
`published_at` datetime DEFAULT NULL,
`source` varchar(255) DEFAULT NULL,
`title` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`url` varchar(511) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UK_1dmji5m90xaiy84vttgkvsub2` (`hash`),
KEY `index_news_source` (`source`),
KEY `index_news_creation_date` (`creation_date`)
) ENGINE=InnoDB AUTO_INCREMENT=30887718 DEFAULT CHARSET=latin1
And a join table to tag news belonging to some popular names:
CREATE TABLE `star_news` (
`stars_id` bigint(20) NOT NULL,
`news_id` bigint(20) NOT NULL,
PRIMARY KEY (`stars_id`,`news_id`),
KEY `FK4eqjn8at6h4d9335q1plxkcnl` (`news_id`),
CONSTRAINT `FK1olc51y8amp8op1kbmx269bac` FOREIGN KEY (`stars_id`) REFERENCES `star` (`id`),
CONSTRAINT `FK4eqjn8at6h4d9335q1plxkcnl` FOREIGN KEY (`news_id`) REFERENCES `news` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Here is my query to return the latest news
SELECT DISTINCT n.*
FROM news n
JOIN star_news sn
ON n.id = sn.news_id
WHERE sn.stars_id IN (1234, 12345)
ORDER BY n.creation_date DESC
LIMIT 2;
Explain:
+----+-------------+-------+------------+--------+-------------------------------------+---------+---------+-----------------------+------+----------+-----------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+--------+-------------------------------------+---------+---------+-----------------------+------+----------+-----------------------------------------------------------+
| 1 | SIMPLE | sn | NULL | range | PRIMARY,FK4eqjn8at6h4d9335q1plxkcnl | PRIMARY | 8 | NULL |196225| 100.00 | Using where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | n | NULL | eq_ref | PRIMARY | PRIMARY | 8 | cosmos_dev.sn.news_id | 1 | 100.00 | NULL |
+----+-------------+-------+------------+--------+-------------------------------------+---------+---------+-----------------------+------+----------+-----------------------------------------------------------+
This query takes 20 seconds on my machine. If I remove the order by clause it returns in sub milli second. How do I make the order by run faster?
I tried using force index on creation_date since its an indexed column, but it worsened the performance
First, write the query as:
SELECT n.*
FROM news n
WHERE EXISTS (SELECT 1
FROM star_news sn
WHERE n.id = sn.news_id AND
sn.stars_id IN (1234, 12345)
)
ORDER BY n.creation_date DESC
LIMIT 2 ;
This eliminates the outer SELECT DISTINCT, which should help.
Then, create an index on star_news(news_id, stars_id). This might also take advantage of an index on news(creation_date desc, id).
So you have 196k news articles relating to those 2 stars? The explain extra tells you what is happening:
Using where; Using index; Using temporary; Using filesort
MySQL is creating a temporary file and sorting it to satisfy the order by because it could not use an index that would facilitate the join AND the ordering of articles by date.

EXPLAIN Shows "DEPENDENT SUBQUERY" and very slow after migrating from MariaDB to MySQL

I was migrating from my old MariaDB 10.0 database to new Google Cloud Sql with Mysql 5.7 with mysqldump method. After migrating, i got some very slow query regarding with WHERE EXISTS statement.
I tried to EXPLAIN my query on both my old DB and new DB and it explained different result. Since it using dump, i am assuming that no changes with the table indexes. This is the query that i wanted to run
SELECT * FROM detitem
where exists (select 1 from detlayanan
where detitem.iddetlayanan = detlayanan.id
and detlayanan.layanan_idlayanan='LYN15176176101503')
the EXPLAIN from old DB
+------+-------------+------------+------+------------------------------------+----------------------------+---------+--------------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+------------+------+------------------------------------+----------------------------+---------+--------------------------------+-------+--------------------------+
| 1 | PRIMARY | detlayanan | ref | PRIMARY,fk_detlayanan_layanan1_idx | fk_detlayanan_layanan1_idx | 22 | const | 11030 | Using where; Using index |
| 1 | PRIMARY | detitem | ref | FK_detitem_detlayanan | FK_detitem_detlayanan | 52 | citridia_sinadme.detlayanan.id | 1 | |
+------+-------------+------------+------+------------------------------------+----------------------------+---------+--------------------------------+-------+--------------------------+
and the EXPLAIN from new DB
+----+--------------------+------------+------------+--------+------------------------------------+---------+---------+---------------------------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+------------+------------+--------+------------------------------------+---------+---------+---------------------------------------+---------+----------+-------------+
| 1 | PRIMARY | detitem | NULL | ALL | NULL | NULL | NULL | NULL | 2079094 | 100.00 | Using where |
| 2 | DEPENDENT SUBQUERY | detlayanan | NULL | eq_ref | PRIMARY,fk_detlayanan_layanan1_idx | PRIMARY | 52 | citridia_sinadme.detitem.iddetlayanan | 1 | 5.00 | Using where |
+----+--------------------+------------+------------+--------+------------------------------------+---------+---------+---------------------------------------+---------+----------+-------------+
The new one is doing Full-table scan even there is index exist. Am i missing something here?
Here is the "detlayanan" table
CREATE TABLE `detlayanan` (
`transaksi_idtransaksi` varchar(40) NOT NULL,
`layanan_idlayanan` varchar(20) NOT NULL,
`nama_layanan` varchar(255) DEFAULT NULL,
`jumlah_beli` float DEFAULT NULL,
`id` varchar(50) NOT NULL,
`harga` decimal(20,2) DEFAULT '0.00',
`hargatotal` decimal(20,2) DEFAULT '0.00',
`luas_p` double(255,2) DEFAULT '0.00',
`luas_l` double(255,2) DEFAULT '0.00',
`luas_q` double(255,2) DEFAULT '0.00',
`keterangan` varchar(255) DEFAULT '',
`iddeposit` varchar(255) DEFAULT NULL,
`posisi` tinyint(4) DEFAULT '1',
`idworkshop` varchar(60) DEFAULT NULL,
`is_wsot` tinyint(4) DEFAULT '0',
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP,
`total_bersih` varchar(20) DEFAULT '0',
`total_min_order` decimal(20,2) DEFAULT '0.00',
`kondisi_barang` text,
PRIMARY KEY (`id`),
KEY `fk_detlayanan_layanan1_idx` (`layanan_idlayanan`),
KEY `fk_detlayanan_deposit` (`iddeposit`),
KEY `transaksi_idtransaksi` (`transaksi_idtransaksi`),
CONSTRAINT `detlayanan_ibfk_1` FOREIGN KEY (`transaksi_idtransaksi`) REFERENCES `transaksi` (`idtransaksi`),
CONSTRAINT `fk_detlayanan_layanan1` FOREIGN KEY (`layanan_idlayanan`) REFERENCES `layanan` (`idlayanan`) ON DELETE NO ACTION ON UPDATE NO ACTION
)
And here is the "detitem" table
CREATE TABLE `detitem` (
`item_iditem` varchar(20) NOT NULL,
`layanan_idlayanan` varchar(255) NOT NULL,
`jumlah_item` int(255) DEFAULT NULL,
`transaksi_idtransaksi` varchar(255) NOT NULL,
`id` varchar(50) NOT NULL,
`iddetlayanan` varchar(50) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP,
`hapus` tinyint(4) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `FK_detitem_item` (`item_iditem`),
KEY `FK_detitem_layanan` (`layanan_idlayanan`),
KEY `FK_detitem_transaksi` (`transaksi_idtransaksi`),
KEY `FK_detitem_detlayanan` (`iddetlayanan`),
CONSTRAINT `FK_detitem_detlayanan` FOREIGN KEY (`iddetlayanan`) REFERENCES `detlayanan` (`id`),
CONSTRAINT `FK_detitem_item` FOREIGN KEY (`item_iditem`) REFERENCES `item` (`iditem`),
CONSTRAINT `FK_detitem_layanan` FOREIGN KEY (`layanan_idlayanan`) REFERENCES `layanan` (`idlayanan`),
CONSTRAINT `FK_detitem_transaksi` FOREIGN KEY (`transaksi_idtransaksi`) REFERENCES `transaksi` (`idtransaksi`)
)
I expect the "rows" on explain stay small, in the old DB, "rows" stays at very low even in another table. but in the new DB it can shows up to million.
UPDATE
After some research, actually i must explicitly add some indexed column in the where statement to accompany EXISTS statement. so the query would be like this
SELECT * FROM detitem WHERE
<indexed column> in (<some id's>)
AND EXISTS ( SELECT 1 FROM detlayanan WHERE detitem.iddetlayanan =
detlayanan.id AND detlayanan.layanan_idlayanan = 'LYN15176176101503' )
apparently MySQL doing full scan on detitem table to check the subquery values is exists or not, when some indexed column are declared, sql does not need to did that. Also this case were found in MySQL.
some references:
https://mariadb.com/kb/en/library/exists-to-in-optimization/
This looks like a case where MariaDB's Optimizer is a step or two ahead of MySQL's.
See if this works well:
SELECT i.*
FROM ( SELECT id
FROM detlayanan
WHERE layanan_idlayanan = 'LYN15176176101503'
) AS x
JOIN detitem AS i ON x.id = i.iddetlayanan
I think it will work well in both servers.
Simpler yet:
SELECT i.*
FROM detlayanan AS lay
JOIN detitem AS i ON lay.id = i.iddetlayanan
WHERE lay.layanan_idlayanan = 'LYN15176176101503'

MySQL Query takes too long to execute

I have a table
CREATE TABLE tb_incident (
id int(10) unsigned NOT NULL AUTO_INCREMENT,
what tinyint(4) NOT NULL,
when bigint(20) unsigned NOT NULL,
where tinytext NOT NULL,
where_longitude_latitude point NOT NULL,
who tinyint(4) DEFAULT NULL,
why tinyint(4) DEFAULT NULL,
description text,
created timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (id),
UNIQUE KEY id_UNIQUE (id),
KEY index_main_items (what,when),
SPATIAL KEY location_index (where_longitude_latitude)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
When I try running this query,
SELECT db_incident.distance(point(132.6,15.604436765987833),where_longitude_latitude,'mi') AS "Distance in miles",
X(where_longitude_latitude) AS "Longitude",
Y(where_longitude_latitude) AS "Latitude"
FROM db_incident.tb_incident
WHERE db_incident.distance(point(132.6,15.604436765987833),where_longitude_latitude,'mi') <= 5
LIMIT 100
It takes over 10 minutes to execute, sometimes it gives this error Error Code: 2013. Lost connection to MySQL server during query
My table has over 10 million records. Can someone please take a look and see if there is a way the query can be optimized.
NOTE: db_incident.distance is a function that returns the distance between two points.
Below is the EXPLAIN of the query.
+----+-------------+-------------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+------+---------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | tb_incident | ALL | NULL | NULL | NULL | NULL | 11689629 | Using where |
+----+-------------+-------------+------+---------------+------+---------+------+----------+-------------+
1 row in set (0.00 sec)

MySQL optimization query

i have one MySQL issue. I have to optimize some queries on my website. One of them i have already done, but there are still some which i cannot resolve without your help.
I have a table called "news":
CREATE TABLE IF NOT EXISTS `news` (
`id` int(10) NOT NULL auto_increment,
`edited` smallint(1) NOT NULL default '0',
`site` varchar(30) default NULL,
`foreign_id` varchar(25) default NULL,
`title` varchar(255) NOT NULL,
`text` text NOT NULL,
`image` varchar(255) default NULL,
`horizontal` smallint(1) NOT NULL,
`image_author` varchar(255) default NULL,
`text_author` varchar(255) default NULL,
`lang` varchar(3) NOT NULL,
`link` varchar(255) NOT NULL,
`date` date NOT NULL,
`redirect` smallint(1) NOT NULL,
`parent` int(10) NOT NULL,
`views` int(5) NOT NULL,
`status` smallint(1) NOT NULL,
PRIMARY KEY (`id`),
KEY `lang` (`lang`,`status`),
KEY `date` (`date`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=47122 ;
as you can see i have two indexes: "lang" and "date"
I have tried some combinations of different indexes and this one has produced me the best results ... unfortunately only on my local computer. On the server i still have bad results. I want to say that the database is the same.
query:
SELECT id FROM news WHERE lang = 'en' AND STATUS =1 ORDER BY DATE DESC LIMIT 0, 10
localhost explain:
+----+-------------+-------+-------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | news | index | lang | date | 3 | NULL | 23 | Using where |
+----+-------------+-------+-------+---------------+------+---------+------+------+-------------+
server explain:
+----+-------------+-------+------+---------------+--------+---------+-------------+-------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+--------+---------+-------------+-------+-----------------------------+
| 1 | SIMPLE | news | ref | status | status | 13 | const,const | 15840 | Using where; Using filesort |
+----+-------------+-------+------+---------------+--------+---------+-------------+-------+-----------------------------+
I have looked a lot of other similar topics, but unfortunately i cannot find any solution to work on my server. I will be very glad to here from you some solution with some explanation for that so i can optimize my other queries.
Thanks !
This is your query:
SELECT id
FROM news
WHERE lang = 'en' AND STATUS =1
ORDER BY DATE DESC
LIMIT 0, 10
The best index is one that contains all the fields used in the query (four fields in all). The ordering in the index is by equality conditions in the where clause followed by the order by clause followed by other columns in the select clause.
So, try this index: ndws(leng, status, date, id).

What is the best way to optimze this query with indexes?

I have a table with about 30 million records which I need to perform queries upon. From my reading, I thought that a composite index using leftmost prefixing with all the fields I need to select would be the correct way to do it, but when I run an explain on the query, it's not even using the index.
This is the query:
select distinct email FROM my_table
WHERE `customer_id` IN(278,428,186,40,208,247,59,79,376,73,38,52,68,227)
AND `company_id` = 4
AND `active` = 1
AND `date` > '2012-04-15';
The explain looks like this
+----+-------------+--------+-------+---------------+-------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+-------+---------------+-------+---------+------+----------+-------------+
| 1 | SIMPLE | emails | index | customer_id | email | 772 | NULL | 29296705 | Using where |
+----+-------------+--------+-------+---------------+-------+---------+------+----------+-------------+
These are the fields
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`email` varchar(255) NOT NULL DEFAULT '',
`customer_id` int(10) unsigned DEFAULT NULL,
`company_id` int(10) unsigned NOT NULL,
`active` tinyint(1) unsigned NOT NULL DEFAULT '1',
`date` date DEFAULT NULL
Indexes looks like this
PRIMARY KEY (`id`),
UNIQUE KEY `email` (`email`,`customer_id`),
KEY `customer_id` (`customer_id`,`company_id`,`active`,`date`)
I'm not quite sure what the best way to optimize this is.
MySQL is often fussy about IN on the left side of the index. Try one query for each customer_id and see if that's using your index. You can use the UNION syntax to join them together The other possibility is that MySQL figures it's faster to sift through everything for 10% of rows than to try to use indexes for them.