I have a select query below, what it does is it selects all the products matching a certain attribute from a Virtuemart table. The attribute table is rather large (almost 6000 rows). Is there any way to optimize the query below or are there any other process that might be helpful, I already tried adding indexes to one and even two tables.
SELECT DISTINCT `jos_vm_product`.`product_id`,
`jos_vm_product_attribute`.`attribute_name`,
`jos_vm_product_attribute`.`attribute_value`,
`jos_vm_product_attribute`.`product_id`
FROM (`jos_vm_product`)
RIGHT JOIN `jos_vm_product_attribute`
ON `jos_vm_product`.`product_id` = `jos_vm_product_attribute`.`product_id`
WHERE ((`jos_vm_product_attribute`.`attribute_name` = 'Size')
AND ((`jos_vm_product_attribute`.`attribute_value` = '6.5')
OR (`jos_vm_product_attribute`.`attribute_value` = '10')))
GROUP BY `jos_vm_product`.`product_sku`
ORDER BY CONVERT(`jos_vm_product_attribute`.`attribute_value`, SIGNED INTEGER)
LIMIT 0, 24
Here is the results of the EXPLAIN table:
id select_type table type possible_keys key key_len ref rows Extras
1 SIMPLE jos_vm_product_attribute range idx_product_attribute_name,attribute_value,attribute_name attribute_value 765 NULL 333 Using where; Using temporary; Using filesort
1 SIMPLE jos_vm_product eq_ref PRIMARY PRIMARY 4 shoemark_com_shop.jos_vm_product_attribute.product_id
Any help would be greatly appreciated. Thanks.
Replacing the jos_vm_product_attribute.attribute_name index with a composite index on jos_vm_product_attribute.attribute_name and jos_vm_product_attribute.attribute_value (in that order) should help this query. Currently, it's only using an index in the WHERE condition for jos_vm_product_attribute.attribute_value, but this new index will be usable for both parts of the WHERE condition.
Related
Let's say I have three tables - users, servers and payments. Each user can have multiple servers and each server can have multiple payments. Let's also say I wanted to find the most recent payments and get info about the servers / customers those payments are attached to. Here's a query that could do this:
SELECT *
FROM payments p
JOIN customers c ON p.custID = c.custID
JOIN servers s ON s.serverID = p.serverID
WHERE c.hold = 0
AND c.archive = 0
ORDER BY p.paymentID DESC
LIMIT 10;
The problem is that when I run EXPLAIN on this query I get this:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE c ref PRIMARY,hold_archive hold_archive 3 const,const 28728 Using where; Using index; Using temporary; Using filesort
1 SIMPLE p ref custID custID 5 customers.custID 3 Using where
1 SIMPLE s eq_ref PRIMARY PRIMARY 4 payments.serverID 1 Using index
The problem is that the query takes a while to run. If I remove the ORDER BY it becomes 10x as fast. But I need the ORDER BY. Here's the EXPLAIN when I remove the ORDER BY:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE c ref PRIMARY,hold_archive hold_archive 3 const,const 28728 Using where; Using index
1 SIMPLE p ref custID custID 5 customers.custID 3 Using where
1 SIMPLE s eq_ref PRIMARY PRIMARY 4 payments.serverID 1 Using index
So the big difference here is that "Using temporary" and "Using filesort" are missing from the Extra column.
It seems like the reason, in this case, is that the column I'm doing the ORDER BY on isn't the first column in the EXPLAIN.
Another observation. If I remove one of the WHERE clauses (whilst keeping the ORDER BY) it speeds up similarily, but I need both WHERE's. Here's an example EXPLAIN of that:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE p index custID,serverID PRIMARY 4 NULL 10 Using where
1 SIMPLE c eq_ref PRIMARY,hold_archive PRIMARY 4 payments.custID 1 Using where
1 SIMPLE s eq_ref PRIMARY PRIMARY 4 payments.serverID 1 Using index
Here the ORDER BY column /is/ being done on the first column of the EXPLAIN. But why is MySQL re-arranging the order the tables are JOINed in and how can I make it so it doesn't do that? You can force indexes in MySQL but it doesn't seem like that'd help..
Any ideas?
10x faster -- It can find "any 10 rows" a lot faster than "find all possible rows, sort them, then deliver 10".
Having WHERE and ORDER BY hit different columns is hard to optimize.
What percentage of payments have hold=0 and archive=0? It sounds like a small percentage? How many rows in each table?
Does anything else need INDEX(hold, archive)? If not, get rid of it. It seems to be only causing trouble here.
If hold=0 and archive=0 is common, then you would prefer the execution to go like your 3rd EXPLAIN -- that is scan payments in descending order. With most of them matching the WHERE, it will usually` need to hit not much more than 10 rows before finding 10 matching rows.
Another solution (other than getting rid of the index) is to change JOIN to STRAIGHT_JOIN in the query. This tells the Optimizer that you know better, and payments should be scanned first, customers second. That works well if my previous paragraph applies.
But the query will screw up (by being slow) if, say, you look for archive=1.
select mobile_no,mobile_source_type_id,voter_id,district_id,
constituency_id,tehsil_id,local_election_body_id,panchayat_id,
booth_id,is_dnd
from mobile_numbers2
where mobile_no not in (
SELECT mobile_number
from mobile_numbers
)
For this Query it's taking more time.
By using Explain query . It showing below message, How optimize this query.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY mobile_numbers2 ALL NULL NULL NULL NULL 7783355 Using where
2 DEPENDENT SUBQUERY mobile_numbers index idx_mobile_numbers_mobile_number,idx_mobile_no idx_mobile_numbers_mobile_number 48 NULL 49256693 Using where; Using index
since I don't have your database at my disposal, I'm unable to test this query so it may need some tweaking, but this might be faster:
select
mobile_no as mobile_no1,
mobile_numbers.mobile_number as mobile_no2,
mobile_source_type_id,
voter_id,
district_id,
constituency_id,
tehsil_id,
local_election_body_id,
panchayat_id,
booth_id,
is_dnd
from mobile_numbers2
left join mobile_numbers on mobile_numbers.mobile_number = mobile_numbers2.mobile_no
where mobile_no2 IS NULL
A couple of notes:
Subqueries, particularly when paired with 'IN()' are slow
if I were maintaining your database, I would create a single 'mobile_numbers' table and I would reference it from all other tables using it's 'id' column, this would make things cleaner/faster in general.
I have a mysql query that I thought should be using my indexes but still seems to be needing to scan alot of rows (I think).
Here is my query:
SELECT DISTINCT DAY(broadcast_at) AS 'days'
from v3211062009
where month(broadcast_at) = 5 and
year(broadcast_at) = 2012
and deviceid = 337 order by days;
On my table I have an index setup on broadcast_at, deviceid. However the results of a explain on this query looks like:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE v3211062009 ref indx_deviceid,indx_tracking_query indx_tracking_query 4 const **172958** Using where; Using index; Using temporary; Using filesort
I don't understand why it needs to look up so many of the rows. The total amount of rows for this deviceid record is only 184085 so my query seems to be almost looking at all of them just to get the result set. Is the index on broadcast_at not working.
I'm obviously doing something fundamentally wrong but can't figure it out. Changing the order of the columns in my index didn't work.
I don't think MySQL can take advantage of the index on broadcast_at if you use functions on that field.
How does it perform if you do:
SELECT DISTINCT DAY(broadcast_at) AS 'days'
from v3211062009
where broadcast_at >= ('2012-05-01') AND
broadcast_at < ('2012-06-01')
and deviceid = 337 order by days;
Simplifying, I have four tables.
ref_TagGroup (top-level descriptive containers for various tags)
ref_Tag (tags with name and unique tagIDs)
ref_Product
ref_TagMap (TagID,Container,ContainerType)
A fifth table, ref_ProductFamily exists but is not directly part of this query.
I use the ref_TagMap table to map tags to products, but also to map Tags to TagGroups and also to product families. The ContainerType is set to PROD/TAGGROUP/PRODFAM accordingly.
So, I want to return the tag group, tagname and the number of products AND product families that the tag is mapped to...so results like:
GroupName | TagName | TagHitCnt
My question is, why does the first query come back in milliseconds, the second query comes back in milliseconds but the third query (which is just adding an "OR" condition to include both tag to product and tag to family mappings) takes forever (well, over ten minutes anyway...I haven't let it run all night yet.)
QUERY 1:
SELECT ref_taggroup.groupname,ref_tag.tagname,COUNT(DISTINCT IFNULL(ref_product.familyid,ref_product.id + 100000000),ref_product.name) AS 'taghitcnt'
FROM (ref_taggroup,ref_tag,ref_product)
LEFT JOIN ref_tagmap GROUPMAP ON GROUPMAP.containerid=ref_taggroup.groupid
LEFT JOIN ref_tagmap PRODMAP ON PRODMAP.containerid=ref_product.id
WHERE
GROUPMAP.tagid=ref_tag.tagid AND GROUPMAP.containertype='TAGGROUP'
AND
PRODMAP.tagid=ref_tag.tagid AND PRODMAP.containertype='PROD'
GROUP BY tagname
ORDER BY groupname,tagname ;
QUERY 2:
SELECT ref_taggroup.groupname,ref_tag.tagname,COUNT(DISTINCT IFNULL(ref_product.familyid,ref_product.id + 100000000),ref_product.name) AS 'taghitcnt'
FROM (ref_taggroup,ref_tag,ref_product)
LEFT JOIN ref_tagmap GROUPMAP ON GROUPMAP.containerid=ref_taggroup.groupid
LEFT JOIN ref_tagmap PRODFAMMAP ON PRODFAMMAP.containerid=ref_product.familyid
WHERE
GROUPMAP.tagid=ref_tag.tagid AND GROUPMAP.containertype='TAGGROUP'
AND
PRODFAMMAP.tagid=ref_tag.tagid AND PRODFAMMAP.containertype='PRODFAM'
GROUP BY tagname
ORDER BY groupname,tagname ;
QUERY 3:
SELECT ref_taggroup.groupname,ref_tag.tagname,COUNT(DISTINCT IFNULL(ref_product.familyid,ref_product.id + 100000000),ref_product.name) AS 'taghitcnt'
FROM (ref_taggroup,ref_tag,ref_product)
LEFT JOIN ref_tagmap GROUPMAP ON GROUPMAP.containerid=ref_taggroup.groupid
JOIN ref_tagmap PRODMAP ON PRODMAP.containerid=ref_product.id
JOIN ref_tagmap PRODFAMMAP ON PRODFAMMAP.containerid=ref_product.familyid
WHERE
GROUPMAP.tagid=ref_tag.tagid AND GROUPMAP.containertype='TAGGROUP'
AND
((PRODMAP.tagid=ref_tag.tagid AND PRODMAP.containertype='PROD')
OR
(PRODFAMMAP.tagid=ref_tag.tagid AND PRODFAMMAP.containertype='PRODFAM' ))
GROUP BY tagname
ORDER BY groupname,tagname ;
--
To answer a question that may come up, the COUNT Distinct ifnull in the select is designed to return one record for large numbers of products that are grouped into families and one record for each 'standalone' product that isn't in a family as well. This code works well in other queries.
I've tried doing a UNION on the first two queries, and that works and comes back very quickly, but it's not practical for other reasons that I won't go into here.
What is the best way to do this? What am I doing wrong?
Thanks!
ADDING EXPLAIN OUTPUT
QUERY1
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE GROUPMAP ALL 5640 Using where; Using temporary; Using filesort
1 SIMPLE ref_tag ref PRIMARY PRIMARY 4 lsslave01.GROUPMAP.tagid 1 Using index
1 SIMPLE ref_taggroup ref PRIMARY PRIMARY 4 lsslave01.GROUPMAP.containerid 3 Using index
1 SIMPLE PRODMAP ALL 5640 Using where; Using join buffer
1 SIMPLE ref_product eq_ref PRIMARY PRIMARY 4 lsslave01.PRODMAP.containerid 1
QUERY2
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE GROUPMAP ALL 5640 Using where; Using temporary; Using filesort
1 SIMPLE ref_tag ref PRIMARY PRIMARY 4 lsslave01.GROUPMAP.tagid 1 Using index
1 SIMPLE ref_taggroup ref PRIMARY PRIMARY 4 lsslave01.GROUPMAP.containerid 3 Using index
1 SIMPLE PRODFAMMAP ALL 5640 Using where; Using join buffer
1 SIMPLE ref_product ref FixtureType FixtureType 5 lsslave01.PRODFAMMAP.containerid 39 Using where
QUERY3
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE GROUPMAP ALL 5640 Using where; Using temporary; Using filesort
1 SIMPLE ref_tag ref PRIMARY PRIMARY 4 lsslave01.GROUPMAP.tagid 1 Using index
1 SIMPLE ref_taggroup ref PRIMARY PRIMARY 4 lsslave01.GROUPMAP.containerid 3 Using index
1 SIMPLE PRODMAP ALL 5640 Using join buffer
1 SIMPLE PRODFAMMAP ALL 5640 Using where; Using join buffer
1 SIMPLE ref_product eq_ref PRIMARY,FixtureType PRIMARY 4 lsslave01.PRODMAP.containerid 1 Using where
enter code here
One more update for anyone who is interested:
I finally let the third query above run to completion. It took right around 1000 seconds. Dividing this time by the time it takes each of the queries (1 or 2) to run, we get a number around 6000...which is very close to the size of the ref_tagmap table that we're using in our dev environment (much larger in production). So, it looks like we're running one query against each record in that table...but I still can't see why.
Any help would be much appreciated...and I mean seriously, seriously appreciated.
This is less an "answer" than a couple of observations/suggestions.
First, I'm curious whether you could GROUP BY on an integer ID instead of the tag name? I'd change the ref_TagMap.containertype field to hold tinyint enumerated values representing the three possible values of TAGGROUP, PROD and PRODFAM. An indexed tinyint field should be slightly faster than an index of string values. It probably won't make much difference though because it's the second conditional in the join clause and there isn't that much spread in the indexed values anyway.
Next is the observation/reminder that when the first half of an OR statement evaluates to FALSE often, then you're making MySQL evaluate both halves of the conditional every time. So you want to put the condition most likely to evaluate to TRUE first (aka prior to the OR).
I doubt either of those two issues are your real problem... though the issue in the second paragraph may play a part. Seems like the quickest way to a performant version of query 3 may be to simply populate a temp table with the results from the first two queries and pull from that temp table to get the results you're looking for from the third. Perhaps in doing so you'll discover why that third query is so slow.
EXPLAIN EXTENDED SELECT `board` . *
FROM `board`
WHERE `board`.`category_id` = '5'
AND `board`.`board_id` = '0'
AND `board`.`display` = '1'
ORDER BY `board`.`order` ASC
The output of the above query is
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE board ref category_id_2 category_id_2 9 const,const,const 4 100.00 Using where
I'm a little confused by this because I have an index that contains the columns that I'm using in the same order they're used in the query...:
category_id_2 BTREE No No
category_id 33 A
board_id 33 A
display 33 A
order 66 A
The output of EXPLAIN can sometimes be misleading.
For instance, filesort has nothing to do with files, using where does not mean you are using a WHERE clause, and using index can show up on the tables without a single index defined.
Using where just means there is some restricting clause on the table (WHERE or ON), and not all record will be returned. Note that LIMIT does not count as a restricting clause (though it can be).
Using index means that all information is returned from the index, without seeking the records in the table. This is only possible if all fields required by the query are covered by the index.
Since you are selecting *, this is impossible. Fields other than category_id, board_id, display and order are not covered by the index and should be looked up.
It is actually using index category_id_2.
It's using the index category_id_2 properly, as shown by the key field of the EXPLAIN.
Using where just means that you're selecting only some rows by using the WHERE statement, so you won't get the entire table back ;)