Below query taking one sec to execute. I need to optimize following. I have 10 query in my page, so its takes more than 15 sec. loading the page. I need help to optimize the query.
SELECT p.mhSlug, p.cgSlug, p.CatDescription, p.cSlug, COUNT(DISTINCT(p.RootModelNumber)) AS prodCount
FROM tbl_products AS p
STRAIGHT_JOIN ModelPrice AS mp ON (mp.product_id = p.id AND mp.Available = 1) STRAIGHT_JOIN Brand_Data AS bd ON bd.pkID = p.pkBrand
STRAIGHT_JOIN tbl_store_brands AS sb ON sb.categoryID = p.pkCategory AND sb.brandID = p.pkBrand
LEFT JOIN tbl_filters AS f ON (f.SKU = p.SKU)
WHERE mhSlug = 'plumbing' AND p.cgSlug = 'bathroom-fixtures' AND mp.Available = 1
GROUP BY p.cSlug
ORDER BY p.cSlug
Here EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE p ref PRIMARY,pkBrand,cgslug,mhslug,pkCategory,CategoryrBrand,mhcgc,mhcg cgslug 228 const 41164 Using where; Using filesort
1 SIMPLE mp ref product_id,product_idAvail product_id 9 ekidtv_uakc.p.id 1 Using where
1 SIMPLE bd eq_ref PRIMARY,id PRIMARY 4 ekidtv_uakc.p.pkBrand 1 Using index
1 SIMPLE sb ref brandID,categoryID,brandCat,CatBrand brandCat 10 ekidtv_uakc.bd.pkID,ekidtv_uakc.p.pkCategory 1 Using where; Using index
1 SIMPLE f ref SKU,SKUSortField SKU 303 ekidtv_uakc.p.SKU 2 Using index
Please help any body. I have struggle last two days.
Any body able to help me. I did not any improvement.
You need to create or review the indexes on your tables. See indexes documentation for mysql.
I dont see anything wrong on your query.
Related
Currently I am facing a rather slow query on a website, which also slows down the server on more traffic. How can I rewrite the query or what index can I write to avoid "Using temporary; Using filesort"? Without "order by" everything works fast, but without the wanted result/order.
SELECT cams.name, models.gender, TIMESTAMPDIFF(YEAR, models.birthdate, CURRENT_DATE) AS age, lcs.viewers
FROM cams
LEFT JOIN cam_tags ON cams.id = cam_tags.cam_id
INNER JOIN tags ON cam_tags.tag_id = tags.id
LEFT JOIN model_cams ON cams.id = model_cams.cam_id
LEFT JOIN models ON model_cams.model_id = models.id
LEFT JOIN latest_cam_stats lcs ON cams.id = lcs.cam_id
WHERE tags.name = '?'
ORDER BY lcs.time_stamp_id DESC, lcs.viewers DESC
LIMIT 24 OFFSET 96;
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
tags
NULL
const
PRIMARY,tags_name_uindex
tags_name_uindex
766
const
1
100
Using temporary; Using filesort
1
SIMPLE
cam_tags
NULL
ref
PRIMARY,cam_tags_cams_id_fk,cam_tags_tags_id_fk
cam_tags_tags_id_fk
4
const
75565047
100
Using where
1
SIMPLE
cams
NULL
eq_ref
PRIMARY
PRIMARY
4
cam_tags.cam_id
1
100
NULL
1
SIMPLE
model_cams
NULL
eq_ref
model_platforms_platforms_id_fk
model_platforms_platforms_id_fk
4
cam_tags.cam_id
1
100
NULL
1
SIMPLE
models
NULL
eq_ref
PRIMARY
PRIMARY
4
model_cams.model_id
1
100
NULL
1
SIMPLE
lcs
NULL
eq_ref
PRIMARY,latest_cam_stats_cam_id_time_stamp_id_viewers_index
PRIMARY
4
cam_tags.cam_id
1
100
NULL
There are many cases where it is effectively impossible to avoid "using temporary, using filesort".
"Filesort" does not necessarily involve a "file"; it is often done in RAM. Hence performance may not be noticeably hurt.
That said, I will assume your real question is "How can this query be sped up?".
Most of the tables are accessed via PRIMARY or "eq_ref" -- all good. But the second table involves touching an estimated 75M rows! Often that happens as the first table, not second. Hmmmm.
Sounds like cam_tags is a many-to-many mapping table? And it does not have any index starting with name? See this for proper indexes for such a table: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table
Since the WHERE and ORDER BY reference more than one table, it is essentially impossible to avoid "using temporary, using filesort".
Worse than that, it needs to find all the ones with "name='?'", sort the list, skip 96 rows, and only finally deliver 24.
I'm running a query like the one below (where '70' is an example, cause I use the same query in my PHP code only changing that value for every "clasificacion") and I wanted to optimize it, so I used EXPLAIN:
EXPLAIN SELECT TE.PK_ID_TAREA_EMPRESA AS ID_EMPRESA, TE.NOMBRE AS NOMBRE_EMPRESA, TCA.PK_ID_TAREA_CATEGORIA AS ID_CATEGORIA, TCA.NOMBRE AS NOMBRE_CATEGORIA, T.TIPO AS TIPO, T.CLIENTE AS CLIENTE, T.PETICION AS PETICION, T.FACTURABLE AS FACTURABLE, TCOM.COMENTARIO AS COMENTARIO, TCOM.PENDIENTE AS PENDIENTE FROM TAREA_COMENTARIO TCOM
INNER JOIN TAREA T
ON TCOM.TAREA_TIPO = T.TIPO
AND TCOM.TAREA_CLIENTE = T.CLIENTE
AND TCOM.TAREA_PETICION = T.PETICION
INNER JOIN TAREA_EMPRESA_ TE
ON TCOM.FK_ID_TAREA_EMPRESA = TE.PK_ID_TAREA_EMPRESA
INNER JOIN TAREA_CATEGORIA_ TCA
ON TCOM.FK_ID_TAREA_CATEGORIA = TCA.PK_ID_TAREA_CATEGORIA
INNER JOIN TAREA_CLASIFICACION_ TCL
ON TCOM.FK_ID_TAREA_CLASIFICACION = TCL.PK_ID_TAREA_CLASIFICACION
WHERE TCOM.FK_ID_TAREA_CLASIFICACION = 70
GROUP BY TCOM.FK_ID_TAREA_EMPRESA, TCOM.FK_ID_TAREA_CATEGORIA, TCOM.FK_ID_TAREA_CLASIFICACION, TCOM.TAREA_TIPO, TCOM.TAREA_CLIENTE, TCOM.TAREA_PETICION;
EXPLAIN returns me this:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE TCL const PRIMARY PRIMARY 4 const 1 Using index; Using temporary; Using filesort
1 SIMPLE TCOM ref PRIMARY,FK_COMENTARIO_CLASIFICACION,FK_COMENTARIO_EMPRESA,FK_COMENTARIO_CATEGORIA FK_COMENTARIO_CLASIFICACION 4 const 18
1 SIMPLE TCA eq_ref PRIMARY PRIMARY 4 FACTURACION_WEB_DEV.TCOM.FK_ID_TAREA_CATEGORIA 1
1 SIMPLE TE ALL PRIMARY (NULL) (NULL) (NULL) 3 Using where
1 SIMPLE T ref TIPO,CLIENTE,PETICION CLIENTE 768 FACTURACION_WEB_DEV.TCOM.TAREA_CLIENTE 76 Using where
That "type ALL" on TE (TAREA_EMPRESA_) is what can't stop bothering me cause it doesn't make sense. I got indexes for every single column (that PRIMARY included), so I don't know why it's not using the index when joining both tables.
This is what I got on TE:
Columns:
Indexes:
Any ideas? Thanks in advance!
EDIT:
TAREA_EMPRESA_ contains this:
PK_ID_TAREA_EMPRESA NOMBRE ORDEN FECHA_BAJA
1 CR ENERGIA 3 (NULL)
2 SPAIRAL COMMERCE 2 (NULL)
3 KNET COMUNICACIONES 4 (NULL)
4 IR SOLUCIONES 1 (NULL)
Please consider the following query
SELECT * FROM PC_SMS_OUTBOUND_MESSAGE AS OM
JOIN MM_TEXTOUT_SERVICE AS TOS ON TOS.TEXTOUT_SERVICE_ID = OM.SERVICE_ID
JOIN PC_SERVICE_NUMBER AS SN ON OM.TO_SERVICE_NUMBER_ID = SN.SERVICE_NUMBER_ID
JOIN PC_SUBSCRIBER AS SUB ON SUB.SERVICE_NUMBER_ID = SN.SERVICE_NUMBER_ID
JOIN MM_CONTACT CON ON CON.SUBSCRIBER_ID = SUB.SUBSCRIBER_ID
--AND CON.MM_CLIENT_ID = 1
AND OM.CLIENT_ID= 1
AND OM.CREATED>='2013-05-08 11:47:53' AND OM.CREATED<='2014-05-08 11:47:53'
ORDER BY OM.SMS_OUTBOUND_MESSAGE_ID DESC LIMIT 50
To get the dataset I require I need to filter on the (commented out) CONTACTS client_id as well as the OUTBOUND_MESSAGES client_id but this is what changes the performance from milliseconds to tens of minutes.
Execution plan without "AND CON.MM_CLIENT_ID = 1":
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE OM index FK4E518EAA19F2EA2B,SERVICEID_IDX,CREATED_IDX,CLIENTID_IDX,CL_CR_ST_IDX,CL_CR_STYPE_ST_IDX,SID_TOSN_CL_CREATED_IDX PRIMARY 8 NULL 6741 3732.00 Using where
1 SIMPLE SUB ref PRIMARY,FKA1845E3459A7AEF FKA1845E3459A7AEF 9 mmlive.OM.TO_SERVICE_NUMBER_ID 1 100.00 Using where
1 SIMPLE SN eq_ref PRIMARY PRIMARY 8 mmlive.OM.TO_SERVICE_NUMBER_ID 1 100.00 Using where
1 SIMPLE CON ref FK2BEC061CA525D30,SUB_CL_IDX FK2BEC061CA525D30 8 mmlive.SUB.SUBSCRIBER_ID 1 100.00
1 SIMPLE TOS eq_ref PRIMARY,FKDB3DF298AB3EF4E2 PRIMARY 8 mmlive.OM.SERVICE_ID 1 100.00
Execution plan with "AND CON.MM_CLIENT_ID = 1":
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE CON ref FK2BEC061CA525D30,FK2BEC06134399E2A,SUB_CL_IDX FK2BEC06134399E2A 8 const 18306 100.00 Using temporary; Using filesort
1 SIMPLE SUB eq_ref PRIMARY,FKA1845E3459A7AEF PRIMARY 8 mmlive.CON.SUBSCRIBER_ID 1 100.00
1 SIMPLE OM ref FK4E518EAA19F2EA2B,SERVICEID_IDX,CREATED_IDX,CLIENTID_IDX,CL_CR_ST_IDX,CL_CR_STYPE_ST_IDX,SID_TOSN_CL_CREATED_IDX FK4E518EAA19F2EA2B 9 mmlive.SUB.SERVICE_NUMBER_ID 3 100.00 Using where
1 SIMPLE SN eq_ref PRIMARY PRIMARY 8 mmlive.SUB.SERVICE_NUMBER_ID 1 100.00 Using where
1 SIMPLE TOS eq_ref PRIMARY,FKDB3DF298AB3EF4E2 PRIMARY 8 mmlive.OM.SERVICE_ID 1 100.00
Any suggestions on how to format the above to make it a little easier on the eye would be good.
ID fields are primary keys.
There are indexes on all joining columns.
You may be able to fix this problem by using a subquery:
JOIN (SELECT C.* FROM CONTACTS C WHERE C.USER_ID = 1) ON C.SUBSCRIBER_ID = SUB.ID
This will materialize the matching rows, which could have downstream effects on the query plan.
If this doesn't work, then edit your query and add:
The explain plans for both queries.
The indexes available on the table.
EDIT:
Can you try creating a composite index:
PC_SMS_OUTBOUND_MESSAGE(CLIENT_ID, CREATED, SERVICE_ID, TO_ SERVICE_ID, SMS_OUTBOUND_MESSAGE_ID);
This may change both query plans to start on the OM table with the appropriate filtering, hopefully making the results stable and good.
I've solved the riddle! For my case anyway so I'll share.
This all came down to the join order changing once I added that extra clause, which you can clearly see in the execution plan. When the query is fast, the Outbound Messages are at the top of the plan but when slow (after adding the clause), the Contacts table is at the top.
I think this means that the Outbound Messages index can no longer be utilised for the sorting which causes the dreaded;
"Using temporary; Using filesort"
By simply adding STRAIGHT_JOIN keyword directly after the select I could force the execution plan to join in the order denoted directly by the query.
Happy for anyone with a more intimate knowledge of this field to contradict any of the above in terms of what is actually happening but it definitely worked.
I have a query running on MySQL DB and is very slow.
Is there anyway I can optimize the following
SELECT mcm.merchant_name,
( ( Sum(ot.price) + Sum(ot.handling_charges)
+ Sum(ot.sales_tax_recd)
+ Sum(ot.shipping_cost) - Sum(ot.sales_tax_payable) ) -
Sum(im.break_even_cost) ) AS PL,
ot.merchant_id
FROM order_table ot,
item_master im,
merchant_master mcm
WHERE ot.item_id = im.item_id
AND ot.merchant_id = mcm.merchant_id
GROUP BY mcm.merchant_name
ORDER BY pl DESC
LIMIT 0, 10;
The Above Query is taking more than 200 seconds to execute.
Explain Result:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ot ALL "merchant_id,item_id" NULL NULL NULL 507910 "Using temporary; Using filesort"
1 SIMPLE mcm eq_ref "PRIMARY,merchant_id" PRIMARY 4 stores.ot.merchant_id 1
1 SIMPLE im eq_ref "PRIMARY,item_id" PRIMARY 4 stores.ot.item_id 1
Also, I got Error-1003 when I run EXPLAIN EXTENDED
use mysql explain plan to find out why it is taking so long and then maybe create some indexes or change your code.
Update
Based upon this make sure you have an composite index on the order_table on merchant_id,item_id
For the last couple of days I have been working on normalizing our 600GB database.
I have broken out all redundant data into 4 separate tables plus a main entry table.
All is well so far, but the last step, joining the new tables with the old data records and inserting the new normalized data records into the database.For this I'm using "INSERT INTO SELECT". But not to the problem. If i run this query on the first 100 id's It takes 10sec but if I run it on the first 300 rows it takes several minutes. What can I do to fix this?
SELECT * FROM oldDB.`unNormalized`
INNER JOIN `new_normalized_db`.`keyword` k ON `unNormalized_tabel`.`keyword` = k.`keyword`
INNER JOIN `new_normalized_db`.`project` p ON `unNormalized_tabel`.`awrProject` = p.`project`
INNER JOIN `new_normalized_db`.`searchEngine` s ON `unNormalized_tabel`.`searchEngine` = s.`searchEngine`
INNER JOIN `new_normalized_db`.`urlHash` u ON MD5(`unNormalized_tabel`.`url`) = u.`hash`
WHERE oldDB.`unNormalized_tabel`.`id` < 100
GROUP BY k.`id`, p.`id`, s.`id`,u.`id`
As of right now the old entrys only have a primery key index, should I add a full text index to all the old data columns? Am thinking this could take months on a 600gb un-normalized database? And what about space how much extra space does 4 new indexes take up?
id select_type table type possible_keys key key_len ref rows Extra
------ ----------- ----------------- ------ --------------------------------------------------------------- ------------ ------- -------------------------------- ------ ----------------------------------------------
1 SIMPLE p index (NULL) projectName 42 (NULL) 427 Using index; Using temporary; Using filesort
1 SIMPLE unormalized_tabel range PRIMARY,keyword_url_insDate,keyword,searchEngine,url,awrProject PRIMARY 4 (NULL) 358 Using where; Using join buffer
1 SIMPLE u ref url url 767 oldDB.unormalized_tabel.url 1
1 SIMPLE k index (NULL) keyword 42 (NULL) 107340 Using where; Using index; Using join buffer
1 SIMPLE s index (NULL) searchEngine 42 (NULL) 1155 Using where; Using index; Using join buffer
You can speed up your query by adding indexes.
The first join of k can be faster if there is an index on keyword and also an index on unNormalized_tabel.keyword the same for project p with awrProject and p.project and s.searchEngine and unNormalized_tabel.searchEngine
But the last join will be slow anyway, because a hash must be calculated on query time, which is very slow on much data. What you can do is hash the url before inserting in the unNormalized_tabel, and then add an index on the hash_field.