Optimize SQL subquery for statistics - mysql

Question

I created a simple statistics tool for our user PCs. It records every 5 minutes the state of all of our PCs. And a little frontend gives me a usage chart:
Now with growing data the SQL queries are getting slower and slower and I'm searching a way to optimize it.
This is the structure. As you can see, the table "usage" contains about 6 million records and it uses MySQL InnoDB:
CREATE TABLE IF NOT EXISTS `usage` (
`id` int(11) unsigned NOT NULL,
`host_id` int(10) unsigned NOT NULL,
`time` int(10) unsigned NOT NULL,
`state` enum('LinuxTU','LinuxExt','View','Browser','Idle','Offline') CHARACTER SET latin1 NOT NULL DEFAULT 'Offline'
) ENGINE=InnoDB AUTO_INCREMENT=5963366 DEFAULT CHARSET=utf8;
ALTER TABLE `usage`
ADD PRIMARY KEY (`id`), ADD KEY `host_id` (`host_id`), ADD KEY `time` (`time`);
ALTER TABLE `usage`
MODIFY `id` int(11) unsigned NOT NULL AUTO_INCREMENT,AUTO_INCREMENT=5963366;
The following query takes about 7 seconds to execute. It is the query that gives the data to the screenshot.
/* create pivot table */
SELECT `time`,
SUM(IF(state='LinuxTU', statecount, 0)) AS LinuxTU,
SUM(IF(state='LinuxExt', statecount, 0)) AS LinuxExt,
SUM(IF(state='View', statecount, 0)) AS View,
SUM(IF(state='Browser', statecount, 0)) AS Browser
FROM (
/* get data from last 24h grouped by state */
SELECT `time`, `state`, COUNT(`state`) statecount
FROM `usage` u
/* group by time to get every 5 minutes
group by state to get the state counter */
GROUP BY `time`, `state`
HAVING `time` > 1441271078 AND `time` < 1441357478
) AS s
GROUP BY `time`
ORDER BY `time` ASC
I don't know how to optimize it. Is there something I missed? Or do I need to reorganize the structure? Any Hint?

Answer 1

In addition to moving the time comparison into a where clause, you can get rid of the subquery entirely:
/* create pivot table */
SELECT `time`,
SUM(state = 'LinuxTU') AS LinuxTU,
SUM(state = 'LinuxExt') AS LinuxExt,
SUM(state = 'View') AS View,
SUM(state = 'Browser') AS Browser
FROM usage u
WHERE `time` > 1441271078 AND `time` < 1441357478
GROUP BY `time`
ORDER BY `time` ASC;

Answer 2

I think your problem is on the last
GROUP BY `time`
ORDER BY `time` ASC
because of the subquery your indexes are gone. So, you should find a way to eliminate that.
Do you also have the option to make some processing in the programming language ? Just make the inner select + the variables without sum from the outer select, also add the order and then make the processing in the programming language.
Or must you write this in a query ?

Answer 3

I've found the bottleneck. The problem is the inner query. HAVING seems to be much slower than WHERE. So I tried some different queries and now I got this result:
Takes 7 seconds:
SELECT `time`, `state`, COUNT(`state`) statecount
FROM `usage` u
GROUP BY `time`, `state`
HAVING `time` > 1441271078 AND `time` < 1441357478
Takes 0.1 seconds:
SELECT `time`, `state`, COUNT(`state`) `statecount`
FROM `usage` u
WHERE `time` > 1441271078 AND `time` < 1441357478
GROUP BY `time`, `state`
And gives me the same result. The frontend is now much faster.