GPU Precision issues for relatively small array sizes? - cuda

Question

I have some CUDA code that does some linear algebra to invert a special type of structured matrix. I calculate RMS error using the results of a serialized version of the algorithm. The error grows with problem size to a greater extent that I would expect. Can anyone provide insight as to why this may be the case?
The GPU code is very naive. This is intentional, and I will optimize it very soon - I just wanted a simple baseline kernel that gives the proper results.
__global__ void levinson_durbin_gpu(TYPE *h0_d, TYPE *h_d, TYPE *v_d, TYPE *x_d, TYPE *y_d, int N) //Naive kernel
{
int j = threadIdx.x;
int i;
__shared__ TYPE hn_1[512];
hn_1[j] = h_d[j];
for(i=1; i<N; i++)
{
if(j < i)
{
TYPE hn = h_d[i];
TYPE yn = y_d[i];
__syncthreads();
//Set up temporary arrays, compute inner products
__shared__ TYPE temp[512]; //Temp for hn_1_J_v
__shared__ TYPE temp2[512]; //Temp for hn_1_J_x
__shared__ TYPE temp3[512]; //Temp for hn_1_v
temp[j] = hn_1[j]*v_d[i-j-1];
temp2[j] = hn_1[j]*x_d[i-j-1];
temp3[j] = hn_1[j]*v_d[j];
__syncthreads();
//Three reductions at once
for(unsigned int s=1; s<i; s*=2)
{
int index = 2*s*j;
if((index+s) < i)
{
temp[index] += temp[index+s];
temp2[index] += temp2[index+s];
temp3[index] += temp3[index+s];
}
__syncthreads();
}
TYPE hn_1_J_v = temp[0];
TYPE hn_1_J_x = temp2[0];
TYPE hn_1_v = temp3[0];
TYPE alpha_v = (hn - hn_1_J_v)/(h0_d[0] - hn_1_v);
TYPE alpha_x = (yn - hn_1_J_x)/(h0_d[0] - hn_1_v);
__shared__ TYPE w_v[512];
w_v[j] = v_d[j] - alpha_v*v_d[i-j-1];
__shared__ TYPE w_x[512];
w_x[j] = x_d[j] - alpha_x*v_d[i-j-1];
v_d[j] = w_v[j];
x_d[j] = w_x[j];
if(j == 0)
{
v_d[i] = alpha_v;
x_d[i] = alpha_x;
}
}
__syncthreads();
}
}
The identifier TYPE is either float or double depending on how I compile the code. I'm using 1 block with N threads (again, keeping things naive and simple here). With single precision I see the following results:
N=4: RMS Error = 0.0000000027
N=8: RMS Error = 0.0000001127
N=16: RMS Error = 0.0000008832
N=32: RMS Error = 0.0000009233
N=64: RMS Error = 42.0136776452
N=80: RMS Error = 281371.7533760048
I can't tell if this is an error with my algorithm or some sort of precision issue. If it helps I can show the above results using double precision, the CPU version of the algorithm, or the code that calculates the RMS error. I'm using a GeForce GTX 660 Ti (cc 3.0) GPU. The variable x_d contains the end result.

Answer 1

Thanks to the help from the comments section I was able to solve the problem myself, so I'll document it here in case others experience a similar issue.
The problem indeed was synchronization issue - my use of __syncthreads() within a divergent control flow block. The solution was to break that control flow block into multiple parts and calling __syncthreads() after each part:
__global__ void levinson_durbin_gpu(TYPE *h0_d, TYPE *h_d, TYPE *v_d, TYPE *x_d, TYPE *y_d, int N) //Naive kernel
{
int j = threadIdx.x;
int i;
__shared__ TYPE hn_1[512];
hn_1[j] = h_d[j];
__syncthreads();
//Set up temporary arrays
__shared__ TYPE temp[512]; //Temp for hn_1_J_v
__shared__ TYPE temp2[512]; //Temp for hn_1_J_x
__shared__ TYPE temp3[512]; //Temp for hn_1_v
TYPE hn;
TYPE yn;
for(i=1; i<N; i++)
{
if(j < i)
{
hn = h_d[i];
yn = y_d[i];
//Compute inner products
temp[j] = hn_1[j]*v_d[i-j-1];
temp2[j] = hn_1[j]*x_d[i-j-1];
temp3[j] = hn_1[j]*v_d[j];
}
__syncthreads();
//Have all threads complete this section to avoid synchronization issues
//Three reductions at once
for(unsigned int s=1; s<i; s*=2)
{
int index = 2*s*j;
if((index+s) < i)
{
temp[index] += temp[index+s];
temp2[index] += temp2[index+s];
temp3[index] += temp3[index+s];
}
__syncthreads();
}
if(j < i)
{
TYPE hn_1_J_v = temp[0];
TYPE hn_1_J_x = temp2[0];
TYPE hn_1_v = temp3[0];
TYPE alpha_v = (hn - hn_1_J_v)/(h0_d[0] - hn_1_v);
TYPE alpha_x = (yn - hn_1_J_x)/(h0_d[0] - hn_1_v);
__shared__ TYPE w_v[512];
w_v[j] = v_d[j] - alpha_v*v_d[i-j-1];
__shared__ TYPE w_x[512];
w_x[j] = x_d[j] - alpha_x*v_d[i-j-1];
v_d[j] = w_v[j];
x_d[j] = w_x[j];
if(j == 0)
{
v_d[i] = alpha_v;
x_d[i] = alpha_x;
}
}
__syncthreads();
}
}
N=32: RMS Error = 0.0000009233
N=64: RMS Error = 0.0000027644
N=128: RMS Error = 0.0000058276
N=256: RMS Error = 0.0000117755
N=512: RMS Error = 0.0000237040
what I learned: When you use synchronization mechanisms in CUDA, make sure all threads reach the same barrier point! I feel as though this sort of thing should produce a compiler warning.