SQL: Previous Column empty when setting AVG() - mysql

Question

Ok, I am a little noobie when it comes to SQL. In fact very muchly so, so I apologize if this is self evident.
I am trying to find out 3 things from database (This table is a log of every message sent):
Total Reply Time
Total # of Replies that were Under 10 Mins
Average Reply Time
Here is my SQL:
SELECT
*, SUM(case when tmp.reply_time <= 10 then 1 else 0 end) as under_10_mins,
COUNT(tmp.reply_time) AS total_replies
FROM
(SELECT
TIMESTAMPDIFF(MINUTE, `date`, reply_date) as reply_time
FROM
tme_email_staff_reply sr
JOIN
tme_user u
ON
u.id = sr.staff_id
JOIN
tme_email_message m
ON
m.id = sr.message_id
WHERE
`reply_date` >= '2017-04-01 00:00:00'
AND
`reply_date` < '2017-04-27 00:00:00'
)
AS tmp
Which outputs:
| reply_time | under_10_mins | total_replies |
| 106 | 165 | 375 |
Now, when I add in:
SELECT
*, SUM(case when tmp.reply_time <= 10 then 1 else 0 end) as under_10_mins,
COUNT(tmp.reply_time) AS total_replies
FROM
(SELECT
TIMESTAMPDIFF(MINUTE, `date`, reply_date) as reply_time,
(AVG(TIMESTAMPDIFF(SECOND, `date`, reply_date))/60) AS average_reply_time
FROM
tme_email_staff_reply sr
JOIN
tme_user u
ON
u.id = sr.staff_id
JOIN
tme_email_message m
ON
m.id = sr.message_id
WHERE
`reply_date` >= '2017-04-01 00:00:00'
AND
`reply_date` < '2017-04-27 00:00:00'
)
AS tmp
my response is:
| reply_time | average_reply_time |under_10_mins | total_replies |
| 106 | 149.08626667 | 0 | 1 |
As you can see, the under_10_mins and total_replies fields have changed.
Schema for tables linked:
tme_email_staff_reply:
id | staff_id | message_id | reply_date |
1 | 234,221,001 | 15fg16d5dgw2 | 2017-04-01 09:34:16 |
tme_user
id | username | password | email | dob | gender |
// data omited
tme_email_message
id | thread_id | From | To | subject | message | message_id
// data omited
Can anyone tell me why this is so? and how to fix it?

Answer 1

Why this is so?
Let's see AVG:
AVG([DISTINCT] expr)
Returns the average value of expr. The DISTINCT option can be used to return the average of the distinct values of expr.
If there are no matching rows, AVG() returns NULL.
And doc in 13.19.1 Aggregate (GROUP BY) Function Descriptions also said:
If you use a group function in a statement containing no GROUP BY clause, it is equivalent to grouping on all rows. For more information, see Section 13.19.3, “MySQL Handling of GROUP BY”.
This means in your subquery, you used avg without group by, this will avg all the rows, then return one row in subquery.
How to fix it?
I think you should move avg from subquery to outer query:
SELECT
SUM(case when tmp.reply_time <= 10 then 1 else 0 end) as under_10_mins,
COUNT(tmp.reply_time) AS total_replies,
AVG(average_reply_time) AS average_reply_time
FROM
(SELECT
TIMESTAMPDIFF(MINUTE, `date`, reply_date) as reply_time,
(TIMESTAMPDIFF(SECOND, `date`, reply_date))/60 AS average_reply_time
FROM
tme_email_staff_reply sr
JOIN
tme_user u
ON
u.id = sr.staff_id
JOIN
tme_email_message m
ON
m.id = sr.message_id
WHERE
`reply_date` >= '2017-04-01 00:00:00'
AND
`reply_date` < '2017-04-27 00:00:00'
)
AS tmp

Answer 2

The issue is because, in your nested query, you are referring to nonaggregated columns not named in the GROUP BY clause on a MySQL version under 5.7.5. See documentation, notice that: The server is free to choose any value from each group.
MySQL < 5.7.5 allow this syntax but has special behaviour (your case):
MySQL extends the standard SQL use of GROUP BY so that the select list can refer to nonaggregated columns not named in the GROUP BY clause. You can use this feature to get better performance by avoiding unnecessary column sorting and grouping. However, this is useful primarily when all values in each nonaggregated column not named in the GROUP BY are the same for each group. The server is free to choose any value from each group, so unless they are the same, the values chosen are indeterminate. Furthermore, the selection of values from each group cannot be influenced by adding an ORDER BY clause. Result set sorting occurs after values have been chosen, and ORDER BY does not affect which values within each group the server chooses.
MySQL >= 5.7.5 allow this syntax and checks for functional dependence:
MySQL 5.7.5 and up implements detection of functional dependence. If the ONLY_FULL_GROUP_BY SQL mode is enabled (which it is by default), MySQL rejects queries for which the select list, HAVING condition, or ORDER BY list refer to nonaggregated columns that are neither named in the GROUP BY clause nor are functionally dependent on them.