Get rate of rise from mysql order table but tooooo slow - mysql

Question

I have a product order table in mysql. It's like this:
create table `order`
(productcode int,
quantity tinyint,
order_date timestamp,
blablabla)
then, to get rate of rise, i wrote this query:
SELECT thismonth.productcode,
(thismonth.ordercount-lastmonth.ordercount)/lastmonth.ordercount as riserate
FROM ( (SELECT productcode,
sum(quantity) as ordercount
FROM `order`
where date_format(order_date,'%m') = 12
group by productcode) as thismonth,
(SELECT productcode,
sum(quantity) as ordercount
FROM `order`
where date_format(order_date,'%m') = 11
group by productcode) as lastmonth)
WHERE thismonth.productcode = lastmonth.productcode
ORDER BY riserate;
but it runs about 30s on my pc(200000 records, 200MB(include other fields)).
Are there any way to increase query speed? I already create index for productcode field.
I thought the reason of low performance is 'GROUP BY', is there any different way?
I tried your answers, but all of them seems not work, and I was wondering if there is something wrong with index(it's not me who created them), so I delete all index and re-created them, everything goes fine -- It only takes 3-4s. And difference between my query and yours is not very obvious. But REALLY thanks you guys, I learned a lot :)

Answer 1

Try adding an index on (ORDER_DATE, PRODUCTCODE) and change the query to eliminate the use of the DATE_FORMAT function, as in:
SELECT thismonth.productcode,
(thismonth.ordercount-lastmonth.ordercount)/lastmonth.ordercount as riserate
FROM ( (SELECT productcode,
sum(quantity) as ordercount
FROM `order`
WHERE ORDER_DATE BETWEEN '01-12-2010' AND '31-12-2010'
GROUP BY PRODUCTCODE) as thismonth,
(SELECT productcode,
sum(quantity) as ordercount
FROM `order`
WHERE ORDER_DATE BETWEEN '01-11-2010' AND '30-11-2010'
group by productcode) as lastmonth)
WHERE thismonth.productcode = lastmonth.productcode
ORDER BY riserate;
Share and enjoy.

Answer 2

Given the sheer amount of data you seem to be working with, optimization may be difficult. I would first look at how you are using the order_date field. It should probably be indexed with the product_code field. I also don't think date_format is the best way to get the month out of the date - MONTH(order_date) would almost certainly be faster.
Failing that, if this is a query that is going to be hit many times, I would create a new table for the historical data and fill it with the results of your inner queries. Since it's historical data, you won't need to continually get the the latest data. Since you won't have to calculate the historical data every time you run the query, it will run a lot faster.

Answer 3

#Bob Jarvis' solution might resolve your speed issue. If not, or if you want to try an alternative:
Add update_month to store the month
of update_date
Update the column for existing rows
Add an index on update_month
Create a BEFORE UPDATE trigger to
set the value of update_month on row
updates
Create a BEFORE INSERT trigger to
set the value of update_month on row
inserts
Modify your query accordingly

Answer 4

SELECT
productcode,
(this_month_count - last_month_count) / last_month_count AS riserate
FROM (
SELECT
o.product,
SUM(CASE MONTH(o.order_date) WHEN MONTH(m.date_start) THEN o.quantity END) AS last_month_count,
SUM(CASE MONTH(o.order_date) WHEN MONTH(m.date_end) THEN o.quantity END) AS this_month_count
FROM `order` o
INNER JOIN (
SELECT
CAST('2010-11-01' AS date) AS date_start,
CAST('2010-12-31' AS date) AS date_end
) m ON o.order_date BETWEEN m.date_start AND m.date_end
GROUP BY o.product
) s

Answer 5

Consider using datetime instead of timestamp
If your only reason to use timestamp is to have auto default value on insert and update, use datetime instead and put now() into your inserts and updates or use triggers. Timestamp gives you additional conversion for time zones, but if you don't have clients connecting to your database from different time zones you are just losing time on conversions. This alone should give you 15-30% speed up.
This might be one of rare cases where optimizer can choose wrong index
And productcode index is wrong in this case. Because you are grouping by productcode and using where for other column, which is not very selective, optimizer may think using index for productcode can speed up things. But with this index used it gives you very random scan through index lookup but still with quite big number of rows, instead of faster sequential semi-full scan without it, but with order_date index to limit number of rows scanned. Optimizer simply doesn't know you can expect rows mostly to be sorted by order_date on the disk and not by productcode. Of course to make order_date index work you have to change your query so for every comparison using order_date column name is on one side of the =,<,> or BETWEEN and constant values on the other side, like suggested by Bob Javis in his answer (+1 to him). So you might want to try his query slightly modified, with correected date formats and force use of order_date index - assuming you have it, if not you really should add it with
ALTER TABLE `order` ADD INDEX order_date( order_date );
So the final query should look like:
SELECT thismonth.productcode,
(thismonth.ordercount-lastmonth.ordercount)/lastmonth.ordercount as riserate
FROM ( (SELECT productcode,
sum(quantity) as ordercount
FROM `order` FORCE INDEX( order_date )
WHERE order_date BETWEEN '2010-12-01' AND '2010-12-31'
GROUP BY productcode) as thismonth,
(SELECT productcode,
sum(quantity) as ordercount
FROM `order` FORCE INDEX( order_date )
WHERE order_date BETWEEN '2010-11-01' AND '2010-11-30'
group by productcode) as lastmonth)
WHERE thismonth.productcode = lastmonth.productcode
ORDER BY riserate;
Not using productid index should give you some speed up (full scan should be faster), and using order_date index even more, depending on how many rows satisfy order_date conditions vs all rows in the table.