I am trying to debug a MYSQL Query for its taking time around 4-5 minutes on one server and it fails on another MYSQL server.
The query is like this:
SELECT
*,
TM.tutor_id as tutor_id,
TIMESTAMPDIFF( YEAR, birthdate, CURDATE( ) ) AS age
from
tutor_master as TM
LEFT JOIN category_master as CM
on TM.category = CM.category_id
LEFT JOIN tutor_expected_rate TER
ON FIND_IN_SET( TER.tutor_id, TM.tutor_id ) > 0
LEFT JOIN admin_shortlist_master SHM
ON TM.tutor_id = SHM.tutor_id
AND (SHM.user_auth_id = 'c84258e9c39059a89ab77d846ddab909')
INNER JOIN level_master LM
ON FIND_IN_SET(LM.level_id, TER.level_id) > 0
WHERE
1=1
GROUP BY
TM.tutor_id
order by
TM.is_priority DESC,
TM.tutor_id DESC
LIMIT
0, 10
and MYSQL explain result is like this:
1 SIMPLE TM NULL ALL PRIMARY,tutor_id NULL NULL NULL 27530 100.00 Using temporary; Using filesort
1 SIMPLE SHM NULL ref tutor_id,user_auth_id user_auth_id 257 const 1 100.00 Using where
1 SIMPLE CM NULL eq_ref PRIMARY PRIMARY 4 toprecru_portal_db.TM.category 1 100.00 NULL
1 SIMPLE LM NULL ALL NULL NULL NULL NULL 11 100.00 Using join buffer (Block Nested Loop)
1 SIMPLE TER NULL ALL NULL NULL NULL NULL 13223 100.00 Using where; Using join buffer (Block Nested Loop)
Now my question is, it there any apparent mistake, which I am overlooking to make the query more efficient.
Thanks.
Related
I have a query which is running a bit slow. It takes a table containing the items that result from a search and then gets the categories that contain 1 or more of these items. The categories (~300) are stored in a nested set model with around 4 levels. I need to know the counts at each level (so an item might be in the grand child category, but also needs to be counted in the child and parent categories).
The query to do this is as follows:-
SELECT category_parent.id,
category_parent.depth,
category_parent.name AS item_sub_category,
category_parent.left_index,
category_parent.right_index,
COUNT(DISTINCT item.id) as total
FROM search_enquiries_found
INNER JOIN item ON search_enquiries_found.item_id = item.id
INNER JOIN category ON item.mmg_code = category.mmg_code
INNER JOIN category category_parent ON category.left_index BETWEEN category_parent.left_index AND category_parent.right_index
WHERE search_enquiries_found.search_enquiry_id = 35
AND item.cost_price > 0
GROUP BY category_parent.id,
category_parent.depth,
item_sub_category,
category_parent.left_index,
category_parent.right_index
ORDER BY category_parent.left_index
With around 12k records found this is taking ~1.5 seconds. The explain follows:-
id select_type table partitions type possible_keys key key_len ref rows Extra
1 SIMPLE category NULL ALL left_index NULL NULL NULL 337 Using temporary; Using filesort
1 SIMPLE item NULL ref PRIMARY,id,mmg_code mmg_code 27 em_entaonline.category.mmg_code 43 Using where
1 SIMPLE search_enquiries_found NULL eq_ref search_enquiry_id,item_id,search_enquiry_id_2,search_enquiry_id_relevance search_enquiry_id 8 const,em_entaonline.item.id 1 Using index
1 SIMPLE category_parent NULL ALL left_index NULL NULL NULL 337 Range checked for each record (index map: 0x2)
The important table is the hierachical category table:-
CREATE TABLE `category` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(128) NOT NULL,
`depth` int(11) NOT NULL,
`is_active` tinyint(1) NOT NULL DEFAULT '1',
`left_index` int(4) NOT NULL,
`right_index` int(4) NOT NULL,
`mmg_code` varchar(25) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `left_index` (`left_index`,`right_index`),
UNIQUE KEY `depth` (`depth`,`left_index`,`right_index`),
KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
This is not usefully using any index on the join to get parent categories.
Changing this query to remove most of the group by fields and just using the unique id (in effect the other fields in the group by are not required, but I tend to include them to keep to strict standards) reduces the time taken down to ~0.5 seconds:-
SELECT category_parent.id,
category_parent.depth,
category_parent.name AS item_sub_category,
category_parent.left_index,
category_parent.right_index,
COUNT(DISTINCT item.id) as total
FROM search_enquiries_found
INNER JOIN item ON search_enquiries_found.item_id = item.id
INNER JOIN category ON item.mmg_code = category.mmg_code
INNER JOIN category category_parent ON category.left_index BETWEEN category_parent.left_index AND category_parent.right_index
WHERE search_enquiries_found.search_enquiry_id = 35
AND item.cost_price > 0
GROUP BY category_parent.id
ORDER BY category_parent.left_index
The explain for this is very slightly different:-
id select_type table partitions type possible_keys key key_len ref rows Extra?
1 SIMPLE category NULL ALL left_index NULL NULL NULL 337 Using temporary; Using filesort
1 SIMPLE item NULL ref PRIMARY,id,mmg_code mmg_code 27 em_entaonline.category.mmg_code 43 Using where
1 SIMPLE search_enquiries_found NULL eq_ref search_enquiry_id,item_id,search_enquiry_id_2,search_enquiry_id_relevance search_enquiry_id 8 const,em_entaonline.item.id 1 Using index
1 SIMPLE category_parent NULL ALL PRIMARY,left_index,depth,name NULL NULL NULL 337 Using where; Using join buffer (Block Nested Loop)
Now I have 2 questions.
1 - Why does removing the fields from the GROUP BY (which are completely dependent on the primary key left in the group by) change the EXPLAIN this way, with such a large change in performance.
2 - In either case the performance is poor, with no key used for the join to the parent category table. Any suggestions on improving this?
As a further point, I have tried reversing the order of the joins and forcing it using STRAIGHT_JOIN. In this case this further improves the performance (~0.2 seconds), but this is an unusual situation (as it is returning all the records from the parent category table - most times the search will return far less records hence far less categories), so I think this will slow things down in more normal situations, and it still isn't using indexes on some of the joins. It would be good if MySQL could decide which way to join the tables depending on the result sets.
SELECT category_parent.id,
category_parent.depth,
category_parent.name AS item_sub_category,
category_parent.left_index,
category_parent.right_index,
COUNT(DISTINCT item.id) as total
FROM category category_parent
STRAIGHT_JOIN category ON category.left_index BETWEEN category_parent.left_index AND category_parent.right_index
STRAIGHT_JOIN item ON item.mmg_code = category.mmg_code
STRAIGHT_JOIN search_enquiries_found ON search_enquiries_found.item_id = item.id
WHERE search_enquiries_found.search_enquiry_id = 35
GROUP BY category_parent.id,
category_parent.depth,
item_sub_category,
category_parent.left_index,
category_parent.right_index
ORDER BY category_parent.left_index
id select_type table partitions type possible_keys key key_len ref rows Extra
1 SIMPLE category_parent NULL ALL left_index NULL NULL NULL 337 Using temporary; Using filesort
1 SIMPLE category NULL ALL left_index NULL NULL NULL 337 Range checked for each record (index map: 0x2)
1 SIMPLE item NULL ref PRIMARY,id,mmg_code mmg_code 27 em_entaonline.category.mmg_code 43 Using index
1 SIMPLE search_enquiries_found NULL eq_ref search_enquiry_id,item_id,search_enquiry_id_2,search_enquiry_id_relevance search_enquiry_id 8 const,em_entaonline.item.id 1 Using index
I have the following query:
(
select f.Product, bpsrelation.bps, f.Week ForecastWeek,
f.FC Forecast, `Order qty` quantity,
s.FC Sales, s.Week salesWeek, s.`Plant Code`, f.`APO Location`
from forecastnew f
right outer join bpsrelation on bpsrelation.Product = f.Product
right outer join sales s on bpsrelation.bps = s.BPS and f.Week = s.Week
and s.`PLant Code` = f.`APO Location`
right outer join orders on orders.Product = f.Product
group by s.Week, s.`PLant Code`,s.Customer,s.bps
)
union
(
select f.Product, bpsrelation.bps, f.Week ForecastWeek,
f.FC Forecast, `Order qty` quantity,
s.FC Sales, s.Week salesWeek, s.`Plant Code`, f.`APO Location`
from forecastnew f
left outer join orders on orders.Product = f.Product
left outer join bpsrelation on bpsrelation.Product = f.Product
left outer join sales s on bpsrelation.bps = s.BPS
and f.Week = s.Week and s.`PLant Code` = f.`APO Location`
where f.`APO Location` is null
group by s.Week, f.`APO Location`, s.Customer,s.bps
);
The explain extended is as follows:
1 PRIMARY orders ALL 33335 100.00 Using temporary; Using filesort
1 PRIMARY f ref idx_forecastnew_Product idx_forecastnew_Product 23 demandplaning.orders.Product 103 100.00
1 PRIMARY bpsrelation ref idx_bpsrelation_Product,idx_bpsrelation_bps idx_bpsrelation_Product 23 demandplaning.f.Product 1 100.00
1 PRIMARY s ref idx_sales_BPS idx_sales_BPS 33 demandplaning.bpsrelation.bps 41 100.00 Using where
2 UNION f ALL 329325 100.00 Using where; Using temporary; Using filesort
2 UNION orders ref idx_orders_Product idx_orders_Product 23 demandplaning.f.Product 64 100.00
2 UNION bpsrelation ref idx_bpsrelation_Product idx_bpsrelation_Product 23 demandplaning.f.Product 1 100.00
2 UNION s ref idx_sales_BPS idx_sales_BPS 33 demandplaning.bpsrelation.bps 41 100.00 Using where
UNION RESULT <union1,2> ALL Using temporary
The forecast and the sales table need to be joined by plant code, week and product.
This takes a lot of time. I searched stackoverflow and internet, it is adviced to do this with foreach or cursor. Can anyone help me with that?
Thanks a lot.
Is there any way to optimize this mysql query?
I have about 100K rows in a table.
select komentarai.* FROM (
SELECT MAX(id) AS max_id FROM komentarai WHERE siteid=1 GROUP BY number
) AS tbl
INNER JOIN `komentarai` ON komentarai.id = tbl.max_id ORDER BY komentarai.added DESC LIMIT XX, 10
Here is an explanation of index:
PRIMARY <derived2> ALL NULL NULL NULL NULL 3372 Using temporary; Using filesort
PRIMARY komentarai eq_ref PRIMARY,id PRIMARY 4 tbl.max_id 1 2
DERIVED komentarai ref siteid siteid 1 5695 Using where; Using temporary; Using filesort
Thanks!
Im using PDO Mysql, and made a request to select cheapest offers for a product in my database. It works fine, only problem is it is slow (for 200 offers (and still just 25 to return)) it takes almost a second, which is a lot higher than what I aim.
I'm no expert in SQL, so I seek for your help on this matter. Here is the request and I'll be happy to provide more info if needed :
SELECT
mo.id AS id,
mo.stock AS stock,
mo.price AS price,
mo.promotional_price AS promotional_price,
mo.picture_1 AS picture_1,
mo.picture_2 AS picture_2,
mo.picture_3 AS picture_3,
mo.picture_4 AS picture_4,
mo.picture_5 AS picture_5,
mo.title AS title,
mo.description AS description,
mo.state AS state,
mo.is_new AS is_new,
mo.is_original AS is_original,
c.name AS name,
u.id AS user_id,
u.username AS username,
u.postal_code AS postal_code,
p.name AS country_name,
ra.cache_rating_avg AS cache_rating_avg,
ra.cache_rating_nb AS cache_rating_nb,
GROUP_CONCAT(md.delivery_mode_id SEPARATOR ', ') AS delivery_mode_ids,
GROUP_CONCAT(ri.title SEPARATOR ', ') AS delivery_mode_titles
FROM
mp_offer mo, catalog_product_i18n c,
ref_country_i18n p, mp_offer_delivery_mode md,
ref_delivery_mode r,
ref_delivery_mode_i18n ri, user u
LEFT JOIN mp_user_review_rating_i18n ra
ON u.id = ra.user_id
WHERE (mo.product_id = c.id
AND mo.culture = c.culture
AND mo.user_id = u.id
AND u.country_id = p.id
AND mo.id = md.offer_id
AND md.delivery_mode_id = ri.id
AND mo.culture = ri.culture)
AND (mo.culture = 1
AND p.culture = 1)
AND mo.is_deleted = 0
AND mo.product_id = 60
AND ((u.holiday_start IS NULL)
OR (u.holiday_start = '0000-00-00')
OR (u.holiday_end IS NULL)
OR (u.holiday_end = '0000-00-00')
OR (u.holiday_start > '2012-05-03')
OR (u.holiday_end < '2012-05-03'))
AND mo.stock > 0
GROUP BY mo.id
ORDER BY IF (mo.promotional_price IS NULL,
mo.price,
LEAST(mo.price, mo.promotional_price)) ASC
LIMIT 25 OFFSET 0;
I take the offers for a particular product that have their "culture" set to 1, are not deleted, that have some stock and whose seller is not in holidays. I order by price (promotional_price when there is one).
Is LEAST a slow function?
Here is the output of EXPLAIN :
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE c const PRIMARY,catalog_product_i18n_product,catalog_product_i18n_culture PRIMARY 8 const,const 1 "Using temporary; Using filesort"
1 SIMPLE mo ref PRIMARY,culture,is_deleted,product_id,user_id culture 4 const 3 "Using where with pushed condition"
1 SIMPLE u eq_ref PRIMARY,user_country PRIMARY 4 database.mo.user_id 1 "Using where with pushed condition"
1 SIMPLE p eq_ref PRIMARY,ref_country_i18n_culture PRIMARY 8 database.u.country_id,const 1
1 SIMPLE r ALL NULL NULL NULL NULL 3 "Using join buffer"
1 SIMPLE ra ALL NULL NULL NULL NULL 4
1 SIMPLE md ref PRIMARY,fk_offer_has_delivery_mode_delivery_mode1,fk_offer_has_delivery_mode_offer1 PRIMARY 4 database.mo.id 2
1 SIMPLE ri eq_ref PRIMARY PRIMARY 2 database.md.delivery_mode_id,const 1
Thanks in advance for your help on optimizing this request.
J
You are not making use of ref_delivery_mode table that you have included in from clause. It's getting cause of Cartesian product of tables result.
I'm having real difficulties optimising a MySQL query. I have to use the existing database structure, but I am getting an extremely slow response under certain circumstances.
My query is:
SELECT
`t`.*,
`p`.`trp_name`,
`p`.`trp_lname`,
`trv`.`trv_prosceslevel`,
`trv`.`trv_id`,
`v`.`visa_destcountry`,
`track`.`track_id`,
`track`.`track_datetoembassy`,
`track`.`track_expectedreturn`,
`track`.`track_status`,
`track`.`track_comments`
FROM
(SELECT
*
FROM
`_transactions`
WHERE
DATE(`tr_datecreated`) BETWEEN DATE('2011-07-01 00:00:00') AND DATE('2011-08-01 23:59:59')) `t`
JOIN
`_trpeople` `p` ON `t`.`tr_id` = `p`.`trp_trid` AND `p`.`trp_name` = 'Joe' AND `p`.`trp_lname` = 'Bloggs'
JOIN
`_trvisas` `trv` ON `t`.`tr_id` = `trv`.`trv_trid`
JOIN
`_visas` `v` ON `trv`.`trv_visaid` = `v`.`visa_code`
JOIN
`_trtracking` `track` ON `track`.`track_trid` = `t`.`tr_id` AND `p`.`trp_id` = `track`.`track_trpid` AND `trv`.`trv_id` = `track`.`track_trvid` AND `track`.`track_status` IN ('New','Missing_Info',
'En_Route',
'Ready_Pickup',
'Received',
'Awaiting_Voucher',
'Sent_Client',
'Closed')
ORDER BY `tr_id` DESC
The results of an explain statement on the above is:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 164 Using temporary; Using filesort
1 PRIMARY track ALL status_index NULL NULL NULL 4677 Using where
1 PRIMARY p eq_ref PRIMARY PRIMARY 4 db.track.track_trpid 1 Using where
1 PRIMARY trv eq_ref PRIMARY PRIMARY 4 db.track.track_trvid 1 Using where
1 PRIMARY v eq_ref visa_code visa_code 4 db.trv.trv_visaid 1
2 DERIVED _transactions ALL NULL NULL NULL NULL 4276 Using where
The query times are acceptable until the value of 'Closed' is included in the very last track.track_status IN clause. The length of time is then increased about 10 to 15 times the other queries.
This makes sense as the 'Closed' status refers to all the clients whose transactions have been dealt with, wihich corresponds to about 90% to 95% of the database.
The issue is, is that in some cases, the search is taking about 45 seconds which is rediculous. I'm sure MySQL can do much better than that and it's just my query at fault, even if the tables do have 4000 rows, but I can't work out how to optimise this statement.
I'd be grateful for some advice about where I'm going wrong and how I should be implementing this query to produce a faster result.
Many thanks
Try this:
SELECT t.*,
p.trp_name,
p.trp_lname,
trv.trv_prosceslevel,
trv.trv_id,
v.visa_destcountry,
track.track_id,
track.track_datetoembassy,
track.track_expectedreturn,
track.track_status,
track.track_comments
FROM
_transactions t
JOIN _trpeople p ON t.tr_id = p.trp_trid
JOIN _trvisas trv ON t.tr_id = trv.trv_trid
JOIN _visas v ON trv.trv_visaid = v.visa_code
JOIN _trtracking track ON track.track_trid = t.tr_id
AND p.trp_id = track.track_trpid
AND trv.trv_id = track.track_trvid
WHERE DATE(t.tr_datecreated)
BETWEEN DATE('2011-07-01 00:00:00') AND DATE('2011-08-01 23:59:59')
AND track.track_status IN ('New','Missing_Info','En_Route','Ready_Pickup','Received','Awaiting_Voucher','Sent_Client', 'Closed')
AND p.trp_name = 'Joe' AND p.trp_lname = 'Bloggs'
ORDER BY tr_id DESC