Need a simple solutin to slow query - mysql

Question

I have following query..
SELECT avg(h.price)
FROM `car_history_optimized` h
LEFT JOIN vin_data vd ON (concat(substr(h.vin,1,8),'_',substr(h.vin,10,3))=vd.prefix)
WHERE h.date >='2015-01-01'
AND h.date <='2015-04-01'
AND h.dealer_id <> 2389
AND vd.prefix IN
(SELECT concat(substr(h.vin,1,8),'_',substr(h.vin,10,3))
FROM `car_history_optimized` h
LEFT JOIN vin_data vd ON (concat(substr(h.vin,1,8),'_',substr(h.vin,10,3))=vd.prefix)
WHERE h.date >='2015-03-01'
AND h.date <='2015-04-01'
AND h.dealer_id =2389)
It finds the average market value of a car sold within last 3 months by everyone else other than (2389) but only those car which have the same Make, Model sold by (2389)
can above query be optimized ? it's taking 2 minutes to run for 11 million records..
Thanks

Answer 1

How often will you use that particular "prefix"? If often, then I will direct you toward indexing a 'virtual' column.
Otherwise, you need
INDEX(date) -- for the outer query
INDEX(dealer_id, date) -- for what is now the subquery
Then do the EXISTS as suggested, or use a LEFT JOIN ... WHERE ... IS NULL.
Is date a DATE? or a DATETIME? You may be including an extra day. Suggest this pattern:
WHERE date >= '2015-01-01'
AND date < '2015-01-01' + INTERVAL 3 MONTH

Answer 2

If you want a simple solution, my initial thought is to figure out a way to not have function calls in your joins.
You negatively affect the chance that an index will be helpful.
(concat(substr(h.vin,1,8),'_',substr(h.vin,10,3))=vd.prefix)
Maybe a like statement would be a better idea, however, either approach in a join clause is to be avoided.
Bottom line is your table structure & relationships here leaves room for improvement... If you need the concat because you are avoiding joining intermediate tables, don't -- allow the indexes to be used and it should improve your query performance.
Also, make sure you have indexes.

Answer 3

I suggest 3 things
add a column and index it (avoid the functions in the join)
use an inner join
use EXISTS (...) instead of IN (...)
To "optimize" that query you need to add a column to the table car_history_optimized which contains the result of concat(substr(vin,1,8),'_',substr(vin,10,3)) and this column should be indexed.
Also, use INNER JOIN. In the current query the left outer join is wasted because you require every row of that table to be IN (the subquery) so NULL from that table isn't permitted hence you have the same effect as an inner join.
Use EXISTS instead of IN
SELECT
AVG(h.price)
FROM car_history_optimized h
INNER JOIN vin_data vd ON h.new_column = vd.prefix
WHERE h.`date` >= '2015-01-01'
AND h.`date` <= '2015-04-01'
AND h.dealer_id <> 2389
AND EXISTS (
SELECT
NULL
FROM car_history_optimized cho
WHERE cho.`date` >= '2015-03-01'
AND cho.`date` <= '2015-04-01'
AND cho.dealer_id = 2389
AND vd.prefix = cho.new_column
)
;
By the way:
I assume already have some indexes and those include date and dealer_id
in future avoid using "date" as a column name (it's a reserved word)