Change Data Capture Redshift - mysql

Question

I have a table
DAY 1
ID
amount
DATE
1
10
12-02-2020
2
15
12-02-2020
3
20
12-02-2020
4
25
12-02-2020
I did a sum of the amount on day one which turns out to be 70
Now next day I have few more rows where the amount is UPDATED an APPENDED
New tables looks like this
DAY 2
ID
amount
DATE
1
10
12-02-2020
2
20
13-02-2020
3
20
12-02-2020
4
25
12-02-2020
5
30
13-02-2020
6
35
14-02-2020
Now if you see the ID 2 has new updates amount which is 20 earlier 15
and it has new data from dates 13 and 14 on ID 5 and 6
Can I just run a query where it will only process the changed data and add it to the
previous sum
so like 30+35+5(as only 5 increased from the last value)
total = 70
Mainly to process changed data

Answer 1

This will very much depend on how the historical data will be provided.
This example requires additional Day column in the historical data table AND that you're using a MySQL version that supports LAG() (e.g. MySQL v8+ OR MariaDB 10.3+). Let's assume that it's possible for the historical data table to be like this:
ID
Amount
Date
Day
1
10
2020-02-12
1
2
15
2020-02-12
1
3
20
2020-02-12
1
4
25
2020-02-12
1
1
10
2020-02-12
2
2
20
2020-02-13
2
3
20
2020-02-12
2
4
25
2020-02-12
2
5
30
2020-02-13
2
6
35
2020-02-14
2
.. then maybe a query like this:
SELECT Day,
SUM(amount) AS Total,
SUM(amount)-LAG(SUM(amount)) OVER (ORDER BY Day) AS diff
FROM historical_data
GROUP BY Day
ORDER BY Day;
OR (in for MariaDB):
SELECT Day, Total,
Total-LAG(Total) OVER (ORDER BY Day) AS Diff
FROM
(SELECT Day,
SUM(amount) AS Total
FROM historical_data
GROUP BY Day) A;
This will return result like:
Day
Total
diff
1
70
2
140
70
I was following an example from this site on how to use LAG() to get the row data value above it an using them to subtract the SUM(amount) value for that day.
Here's a demo fiddle of the experiment.