Fastest way to calculate correlation in every row? - mysql

Question

Well, I have a table data of millions of rows. I want to carry out correlation study for every row (from the 1st to the current row minus 1). For e.g. the 1st rows is omitted. The 2nd row's result column is to be supplied with the correlation using the 1st row. The 3rd row's result column is to be supplied with the correlation using the 1st and 2nd row. And so on.
Correlation for the entire table can be calculated using:
SELECT (Count(*)*Sum(x*y)-Sum(x)*Sum(y))/
(sqrt(Count(*)*Sum(x*x)-Sum(x)*Sum(x))*
sqrt(Count(*)*Sum(y*y)-Sum(y)*Sum(y))) AS TotalCorelation FROM Data;
I want to avoid using Joins as much as possible as it takes lots of time, sometimes even timeout error, above 300 seconds). What's the other alternative?
Example table Data Structure:
id, x, y, result
1 , 4, 2, null
2 , 6, 3, -0.2312
3 , 5, 5, 0.42312
4 , 6, 2, -0.5231
5 , 5, 5, 0.22312
6 , 3, 7, -0.2312
7 , 2, 9, 0.42231
8 , 7, 2, 0.32253
9 , 9, 5, 0.32431
id : primary key
x and y : The data
result: correlation

Answer 1

I think this is it:
SELECT d2.ID, d2.x, d2.y, d2.result,
(Count(*)*Sum(d1.x*d1.y)-Sum(d1.x)*Sum(d1.y))/
(sqrt(Count(*)*Sum(d1.x*d1.x)-Sum(d1.x)*Sum(d1.x))*
sqrt(Count(*)*Sum(d1.y*d1.y)-Sum(d1.y)*Sum(d1.y))) AS TotalCorelation
FROM Data d1
RIGHT JOIN Data d2 ON d1.id < d2.id
GROUP BY d2.ID
ORDER BY d2.ID
Without a closed form for calculating correlation of N+1 from N rows, you have to use a quadratic join like this.
I'm assuming that your basic formula is correct. But I'm not sure it is -- when I just run it on the total dataset, I don't get the result 0.32431, I get -0.552773693079.
Here's a linear implementation:
SET #SumX = 0;
SET #SumY = 0;
SET #Count = 0;
SET #SumX2 = 0;
SET #SumY2 = 0;
SET #SumXY = 0;
SELECT id, x, y,
#SumX := #SumX + x AS SumX,
#SumY := #SumY + y AS SumY,
#Count := #Count + 1 AS ct,
#SumX2 := #SumX2 + x*x AS SumX2,
#SumY2 := #SumY2 + y*y AS SumY2,
#SumXY := #SumXY + x*y AS SumXY,
IF(#Count > 1,
(#Count*#SumXY-#SumX*#SumY)/
(sqrt(#Count*#SumX2-#SumX*#SumX)*
sqrt(#Count*#SumY2-#SumY*#SumY)), NULL) AS TotalCorelation
FROM DATA
ORDER BY id
SQLFIDDLE