MySQL SUM Query with Joins and Union All - mysql

Question

Can anybody tell me why the second LEFT JOIN in the query below (previous_spend) is returning double the amount it should?
It returns the correct amount if I remove the first LEFT JOIN, so I assume it's doubling the result due to the first LEFT JOIN, but I'm not sure how to rewrite the query to avoid that.
Any help would be much appreciated.
Update: I've created a slightly easier to understand version here: Here's a live example: http://sqlfiddle.com/#!9/6a9358/1 - as you can see 'current_spend' should return £300.
SELECT
COALESCE(SUM(current_spend.total_spend), 0) AS total_spend,
COALESCE(SUM(previous_spend.total_previous_spend), 0) AS total_previous_spend,
COALESCE(SUM(current_spend.total_spend), 0) - COALESCE(SUM(previous_spend.total_previous_spend), 0) AS total_spend_diff,
100 * (COALESCE(SUM(current_spend.total_spend), 0) - COALESCE(SUM(previous_spend.total_previous_spend), 0)) / COALESCE(SUM(previous_spend.total_previous_spend), 0) AS total_spend_diff_perc
FROM customer_scheme
LEFT JOIN (
SELECT SUM(spend_1 + spend_2) AS total_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2017'
GROUP BY user_id
UNION ALL
SELECT SUM(spend_1 + spend_2 + spend_3 + spend_4 + spend_5 + spend_6 + spend_7 + spend_8 + spend_9 + spend_10 + spend_11 + spend_12) AS total_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2018'
GROUP BY user_id
UNION ALL
SELECT SUM(spend_1 + spend_2 + spend_3 + spend_4 + spend_5 + spend_6 + spend_7 + spend_8 + spend_9 + spend_10 + spend_11 + spend_12) AS total_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2019'
GROUP BY user_id
UNION ALL
SELECT SUM(spend_1 + spend_2) AS total_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2020'
GROUP BY user_id
) as current_spend
ON current_spend.user_id = customer_scheme.user_id
LEFT JOIN
(
SELECT SUM(spend_1 + spend_2) AS total_previous_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2013'
GROUP BY user_id
UNION ALL
SELECT SUM(spend_1 + spend_2 + spend_3 + spend_4 + spend_5 + spend_6 + spend_7 + spend_8 + spend_9 + spend_10 + spend_11 + spend_12) AS total_previous_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2014'
GROUP BY user_id
UNION ALL
SELECT SUM(spend_1 + spend_2 + spend_3 + spend_4 + spend_5 + spend_6 + spend_7 + spend_8 + spend_9 + spend_10 + spend_11 + spend_12) AS total_previous_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2015'
GROUP BY user_id
UNION ALL
SELECT SUM(spend_1 + spend_2) AS total_previous_spend, user_id
FROM customer_spend
WHERE customer_spend.spend_year = '2016'
GROUP BY user_id
) as previous_spend
ON previous_spend.user_id = customer_scheme.user_id
LEFT JOIN user
ON customer_scheme.user_id = user.user_id
WHERE customer_scheme.scheme_id = 36
AND customer_scheme.customer_scheme_access = 'Yes'
AND user.user_deleted_at IS NULL
AND user_type = 'Customer'
AND user.user_status IN (1)
ORDER BY total_spend_diff DESC

Answer 1

The summary of your query is
(customer_scheme LEFT JOIN current_spend) appended with (customer_scheme LEFT JOIN previous_spend)
If you remove the LEFT JOIN with current_spend :
Query would be customer_scheme LEFT JOIN previous_spend
That means, for one user in customer_scheme, you will get one row for previous_spend for 2013, one row for 2014, one row for 2015, one row for 2016
While summing, you will sum all the above said 4 rows.
If you have the LEFT JOIN with current_spend :
For first LEFT JOIN with current_spend, you will get one row for 2017, one for 2018, one for 2019 and one for 2020.
So, for one user_id , you will get 4 rows. Now, you are going to join previous_spend with these 4 rows (where as previously, you will join with only one row). While summing, this makes the difference.
I will try to provide much better insights if you can share an sql fiddle with some test data.

Answer 2

Try this:
SELECT
SUM(IFNULL(current_spend.total_spend, 0)) AS total_spend,
SUM(IFNULL(previous_spend.total_previous_spend, 0)) AS total_previous_spend,
SUM(IFNULL(current_spend.total_spend, 0)) - SUM(IFNULL(previous_spend.total_previous_spend, 0)) AS total_spend_diff,
100 * (SUM(IFNULL(current_spend.total_spend, 0)) - SUM(IFNULL(previous_spend.total_previous_spend, 0))) / SUM(IFNULL(previous_spend.total_previous_spend, 0)) AS total_spend_diff_perc
FROM
customer_scheme
LEFT JOIN
(
SELECT SUM(spend_1 + spend_2 + spend_3 + spend_4 + spend_5 + spend_6 + spend_7 + spend_8 + spend_9 + spend_10 + spend_11 + spend_12) AS total_spend, user_id, customer_spend.spend_year
FROM customer_spend
WHERE customer_spend.spend_year in ('2017','2018','2019','2020')
GROUP BY user_id, customer_spend.spend_year
) as current_spend
ON current_spend.user_id = customer_scheme.user_id
INNER JOIN
(
SELECT SUM(spend_1 + spend_2 + spend_3 + spend_4 + spend_5 + spend_6 + spend_7 + spend_8 + spend_9 + spend_10 + spend_11 + spend_12) AS total_previous_spend, user_id, customer_spend.spend_year
FROM customer_spend
WHERE customer_spend.spend_year in ('2013','2014','2015','2016')
GROUP BY user_id, customer_spend.spend_year
) as previous_spend
ON previous_spend.user_id = customer_scheme.user_id
LEFT JOIN user
ON customer_scheme.user_id = user.user_id
WHERE customer_scheme.scheme_id = 36
AND customer_scheme.customer_scheme_access = 'Yes'
AND user.user_deleted_at IS NULL
AND user_type = 'Customer'
AND user.user_status IN (1)
ORDER BY total_spend_diff DESC;