Slow Mysql Inner joins with multiple OR - mysql

Question

I'm helping a friend with an e-commerce site. He has options for users to select different colours, styles, use and type of the products he's selling. The query the adds the following to the query:
INNER JOIN tbl_coloursProducts col ON ( p.product_id = col.productID AND (col.colourID = 2 OR col.colourID = 3 OR col.colourID = 5 OR col.colourID = 8 OR col.colourID = 10))
INNER JOIN tbl_useProducts tbluse ON ( p.product_id = tbluse.productID AND (tbluse.useID = 15 OR tbluse.useID = 16 OR tbluse.useID = 17 OR tbluse.useID = 18))
INNER JOIN tbl_styleProducts style ON ( p.product_id = style.productID AND (style.styleID = 39 OR style.styleID = 44))
INNER JOIN tbl_typeProducts type ON ( p.product_id = type.productID AND (type.typeID = 46 OR type.typeID = 48 OR type.typeID = 50))
The query loads fast enough when only a few options are selecting, but some users are selecting multiple or each which is causing the query to run in excess of 30 seconds and timing out.
Without altering the table structure is there a better way to optimise the query?
This is the full query:
SELECT *,
p.product_id,
Coalesce((SELECT p2sp.price
FROM ab_product_specials p2sp
WHERE p2sp.product_id = p.product_id
AND p2sp.customer_group_id = '1'
AND ( ( p2sp.date_start = '0000-00-00'
OR p2sp.date_start < Now() )
AND ( p2sp.date_end = '0000-00-00'
OR p2sp.date_end > Now() ) )
ORDER BY p2sp.priority ASC,
p2sp.price ASC
LIMIT 1), p.price) AS final_price,
pd.name AS name,
m.name AS manufacturer,
ss.name AS stock,
(SELECT Avg(r.rating)
FROM ab_reviews r
WHERE p.product_id = r.product_id
GROUP BY r.product_id) AS rating,
(SELECT Count(rw.review_id)
FROM ab_reviews rw
WHERE p.product_id = rw.product_id
GROUP BY rw.product_id) AS review
FROM ab_products p
LEFT JOIN ab_product_descriptions pd
ON ( p.product_id = pd.product_id
AND pd.language_id = '1' )
LEFT JOIN ab_products_to_stores p2s
ON ( p.product_id = p2s.product_id )
LEFT JOIN ab_manufacturers m
ON ( p.manufacturer_id = m.manufacturer_id )
LEFT JOIN ab_stock_statuses ss
ON ( p.stock_status_id = ss.stock_status_id
AND ss.language_id = '1' )
LEFT JOIN ab_products_to_categories p2c
ON ( p.product_id = p2c.product_id )
INNER JOIN tbl_coloursproducts col
ON ( p.product_id = col.productid
AND ( col.colourid = 2
OR col.colourid = 3
OR col.colourid = 5
OR col.colourid = 8
OR col.colourid = 10 ) )
INNER JOIN tbl_useproducts tbluse
ON ( p.product_id = tbluse.productid
AND ( tbluse.useid = 15
OR tbluse.useid = 16
OR tbluse.useid = 17
OR tbluse.useid = 18 ) )
INNER JOIN tbl_styleproducts style
ON ( p.product_id = style.productid
AND ( style.styleid = 39
OR style.styleid = 44 ) )
INNER JOIN tbl_typeproducts type
ON ( p.product_id = type.productid
AND ( type.typeid = 46
OR type.typeid = 48
OR type.typeid = 50 ) )
WHERE p.status = '1'
AND p.date_available <= Now()
AND p2s.store_id = 0
AND p2c.category_id = 131
GROUP BY p.product_id
ORDER BY p.product_id DESC
LIMIT 0, 8
Without the custom bits the query runs fine.

Answer 1

Looking at that query, not sure the ORs are the problem themselves (although you could possibly make the code more compact by using and IN clause for each one). Rather I suspect that selecting more and more options results in more rows being returned. And this is causing problems with the sub queries in the SELECT clause.
Can you try the query with the sub queries removed from the SELECT clause and see the effect that has.
You can remove the sub queries quite easily.
SELECT *,
p.product_id,
Coalesce(sub1.price, p.price) AS final_price,
pd.name AS name,
m.name AS manufacturer,
ss.name AS stock,
sub0.rating,
sub0.review
FROM ab_products p
INNER JOIN
(
SELECT r.product_id,
Avg(r.rating) AS rating,
Count(rw.review_id) AS review
FROM ab_reviews r
GROUP BY r.product_id
) sub0
ON p.product_id = sub0.product_id
LEFT OUTER JOIN
(
SELECT p2sp.product_id,
SUBSTRING_INDEX(GROUP_CONCAT(p2sp.price ORDER BY p2sp.priority ASC, p2sp.price ASC ), ',', 1) AS price
FROM ab_product_specials p2sp
WHERE p2sp.customer_group_id = '1'
AND ( p2sp.date_start = '0000-00-00' OR p2sp.date_start < NOW() )
AND ( p2sp.date_end = '0000-00-00' OR p2sp.date_end > NOW() )
GROUP BY p2sp.product_id
) sub1
ON p.product_id = sub1.product_id
LEFT JOIN ab_product_descriptions pd
ON ( p.product_id = pd.product_id
AND pd.language_id = '1' )
LEFT JOIN ab_products_to_stores p2s
ON ( p.product_id = p2s.product_id )
LEFT JOIN ab_manufacturers m
ON ( p.manufacturer_id = m.manufacturer_id )
LEFT JOIN ab_stock_statuses ss
ON ( p.stock_status_id = ss.stock_status_id
AND ss.language_id = '1' )
LEFT JOIN ab_products_to_categories p2c
ON ( p.product_id = p2c.product_id )
INNER JOIN tbl_coloursproducts col
ON ( p.product_id = col.productid
AND ( col.colourid = 2
OR col.colourid = 3
OR col.colourid = 5
OR col.colourid = 8
OR col.colourid = 10 ) )
INNER JOIN tbl_useproducts tbluse
ON ( p.product_id = tbluse.productid
AND ( tbluse.useid = 15
OR tbluse.useid = 16
OR tbluse.useid = 17
OR tbluse.useid = 18 ) )
INNER JOIN tbl_styleproducts style
ON ( p.product_id = style.productid
AND ( style.styleid = 39
OR style.styleid = 44 ) )
INNER JOIN tbl_typeproducts type
ON ( p.product_id = type.productid
AND ( type.typeid = 46
OR type.typeid = 48
OR type.typeid = 50 ) )
WHERE p.status = '1'
AND p.date_available <= Now()
AND p2s.store_id = 0
AND p2c.category_id = 131
GROUP BY p.product_id
ORDER BY p.product_id DESC
LIMIT 0, 8
As an aside, when you read from ab_product_specials you are checking for the date_start and date_end to be 0000-00-00 (ie, dates), but also comparing them with NOW() which returns a date / time field. Are those fields date or date / time fields?

Answer 2

My first though was to use IN to make the query eaier to read:
INNER JOIN tbl_coloursProducts col
ON p.product_id = col.productID AND col.colourID IN ( 2, 3, 5, 8, 10 )
Then I thought, I wonder if they are dynamically building SQL text to squirt into the database logic?! The optimizer is unlikely to do well at optimizing queries when they are constantly mutating in this way.
Consider a scratch table (pseudo code):
-- One time:
CREATE TABLE SratchColours ( colourID INT NOT NULL UNQIUE );
-- For each query:
DELETE FROM SratchColours;
INSERT INTO SratchColours VALUES ( 2 ), ( 3 ), ( 5 ), ( 8 ), ( 10 );
Now you dynamic list of values simply becomes just another join:
tbl_coloursProducts NATURAL JOIN SratchColours
(or you could use an inner join if you must!)
Now, having one base table for every concurrent user is probably not a great way to scale a system. Therefore, consider how to pass a bag of colourID values to the database logic (say, a stored proc), put them into a table (say, a temporary table), then join from there to your base tables.