CUDA Global Barrier -- Works on Kepler and not Fermi - cuda

Question

The following global barrier works on Kepler K10 and not Fermi GTX580:
__global__ void cudaKernel (float* ref1, float* ref2, int* lock, int time, int dim) {
int gid = blockIdx.x * blockDim.x + threadIdx.x;
int lid = threadIdx.x;
int numT = blockDim.x * gridDim.x;
int numP = int (dim / numT);
int numB = gridDim.x;
for (int t = 0; t < time; ++t) {
// compute # time t
for (int i = 0; i < numP; ++i) {
int idx = gid + i * numT;
if (idx > 0 && idx < dim - 1)
ref2 [idx] = 0.333f * ((ref1 [idx - 1] + ref1 [idx]) + ref1 [idx + 1]);
}
// global sync
if (lid == 0){
atomicSub (lock, 1);
while (atomicCAS(lock, 0, 0) != 0);
}
__syncthreads();
// copy-back # time t
for (int i = 0; i < numP; ++i) {
int idx = gid + i * numT;
if (idx > 0 && idx < dim - 1)
ref1 [idx] = ref2 [idx];
}
// global sync
if (lid == 0){
atomicAdd (lock, 1);
while (atomicCAS(lock, numB, numB) != numB);
}
__syncthreads();
}
}
So, by looking at the output sent back to CPU, I noticed that one thread (either 1st or last thread) escapes the barrier and resumes execution earlier than the others. I'm using CUDA 5.0. number of blocks is also always smaller than number of SMs (in my set of runs).
Any idea why the same code wouldn't work on two architectures? What's new in Kepler that helps this global synchronization?

Answer 1

So I suspect the barrier code itself is probably working the same way. It's what's happening on other data structures not associated with the barrier functionality itself that is at issue, it seems.
Niether Kepler nor Fermi have L1 caches that are coherent with each other. What you have discovered (although it's not associated with your barrier code itself) is that the L1 cache behavior is different between Kepler and Fermi.
In particular, Kepler L1 cache is not in play on global loads as described in the above link, and so the caching behavior is handled at L2 level which is device-wide, and therefore coherent. When a Kepler SMX reads it's global data, it's getting coherent values from L2.
On the other hand, Fermi has L1 caches that also participate in global loads (by default -- although this behavior can be turned off) and the L1 caches as described in the link above are unique to each Fermi SM and are non-coherent with the L1 caches in other SMs. When a Fermi SM reads it's global data, it's getting values from the L1, which may be non-coherent with other L1 caches in other SMs.
This is the difference in "coherency" that you are seeing, of the data you are manipulating before and after the barrier.
As I mentioned, I believe the barrier code itself is probably working the same way on both devices.