Temporary table bogging down database -- how to optimize? - mysql

Question

I apologize in advance if this question is too specific, but I think that it is a fairly typical scenario: join and group bys bogging down the db and the best way to get around it. My specific problem is that I need to create a scoreboard based on:
plays (userid,gameid,score) 40M rows
games (gameid) 100K rows
app_games (appid,gameid) ie, the games are grouped into apps and there's a total score for the app which is the sum on all its associated games <20 rows
The users can play multiple times and their best score for each game is recorded. Formulating the query is easy, I've done several variations but they have a nasty tendency to get locked in "copying temp table" for 30-60 seconds when under load.
What can I do? Are there server variables that I should be tweaking or is there a way to reformulate the query to make it faster? The derived version of the query that I'm using is as follows (minus a user table join to grab the name):
select userID,sum(score) as cumscore from
(select userID, gameID,max(p.score) as score
from play p join app_game ag using (gameID)
where ag.appID = 1 and p.score>0
group by userID,gameID ) app_stats
group by userid order by cumscore desc limit 0,20;
Or as a temp table:
drop table if exists app_stats;
create temporary table app_stats
select userID,gameID,max(p.score) as score
from play p join app_game ag using (gameID)
where ag.appID = 1 and p.score>0
group by userid,gameID;
select userID,sum(score) as cumscore from app_stats group by userid
order by cumscore desc limit 0,20;
I have indexes as follows:
show indexes from play;
+-------+------------+----------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+
| play | 0 | PRIMARY | 1 | playID | A | 38353712 | NULL | NULL | | BTREE | |
| play | 0 | uk_play_uniqueID | 1 | uniqueID | A | 38353712 | NULL | NULL | YES | BTREE | |
| play | 1 | play_score_added | 1 | dateTimeFinished | A | 19176856 | NULL | NULL | YES | BTREE | |
| play | 1 | play_score_added | 2 | score | A | 19176856 | NULL | NULL | | BTREE | |
| play | 1 | fk_playData_game | 1 | gameID | A | 76098 | NULL | NULL | | BTREE | |
| play | 1 | user_hiscore | 1 | userID | A | 650062 | NULL | NULL | YES | BTREE | |
| play | 1 | user_hiscore | 2 | score | A | 2397107 | NULL | NULL | | BTREE | |
+-------+------------+----------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+

Answer 1

I suspect both queries when you create the temp table basically needs to go through all the data in your table (and likewise in your do-everything-at-once query). If you have a lot of data that's just going to take a little while.
I'd maintain a separate table with the ID and total score for each player. Whenever you update the play table, also update the summary table. If they get out of sync, just stop the summary table and re-create the data from the play table. (Or if you already use redis in your infrastructure, you could maintain the summary there -- it has functions to make this particular thing really fast).

Answer 2

Instead of making temporary tables, try making a view instead. You can query against it just like you do with your normal table, but it also updates when any data in the view changes. That's far faster than dropping the table and re-creating it every time.