Why memory prefetch has no impact when transferring from device to host? - cuda

Question

I have the following setup:
constexpr uint32_t N{512};
constexpr uint32_t DATA_SIZE{sizeof(float) * N * N};
__managed__ float ma[N * N];
__managed__ float mb[N * N];
__managed__ float mc[N * N];
__global__ void kernel()
{
for (uint32_t i{0}; i < N * N; ++i)
{
mc[i] = ma[i] + mb[i];
}
}
int main(int argc, char *[])
{
for (uint32_t i{0}; i < N * N; ++i)
{
ma[i] = 1.0f;
mb[i] = 2.0f;
}
int deviceId{};
gpuErrchk(cudaGetDevice(&deviceId));
gpuErrchk(cudaMemPrefetchAsync(ma, DATA_SIZE, deviceId, nullptr));
gpuErrchk(cudaMemPrefetchAsync(mb, DATA_SIZE, deviceId, nullptr));
kernel<<<1, 1>>>();
gpuErrchk(cudaPeekAtLastError());
gpuErrchk(cudaMemPrefetchAsync(mc, DATA_SIZE, cudaCpuDeviceId, nullptr));
gpuErrchk(cudaDeviceSynchronize());
float result{0.0f};
for (uint32_t i{0}; i < N * N; ++i)
{
result += mc[i];
}
return static_cast<int>(result);
}
I compile the code with 03 optimizations. Profiling it with nvprof ./test gives me the following (only the memory part):
==29300== Unified Memory profiling result:
Device "Quadro P1000 (0)"
Count Avg Size Min Size Max Size Total Size Total Time Name
2 1.0000MB 1.0000MB 1.0000MB 2.000000MB 164.9620us Host To Device
20 153.60KB 4.0000KB 1.0000MB 3.000000MB 266.0500us Device To Host
19 - - - - 551.9440us Gpu page fault groups
Total CPU Page faults: 9
The first line - HtoD - is straightforward - there were 2 prefetches for ma and mb arrays 1MB each.
The second line is strange for 2 reasons:
Prefetching was ignored (well, not completely, more on this later)
The total size of the data is 3MB despite the fact that the total array size is 1MB and in cudaMemPrefetchAsync also 1MB was specified.
If I run the same code with prefetching commented out I have the following results:
==30051== Unified Memory profiling result:
Device "Quadro P1000 (0)"
Count Avg Size Min Size Max Size Total Size Total Time Name
20 102.40KB 4.0000KB 508.00KB 2.000000MB 189.9230us Host To Device
29 105.93KB 4.0000KB 512.00KB 3.000000MB 278.4960us Device To Host
24 - - - - 1.311533ms Gpu page fault groups
Total CPU Page faults: 14
As seen in the table prefetching has an impact on the number of transfers - for HtoD it was changed from 2 to 20, and for DtoH it changed from 20 to 29. It also has an impact on performance, but that impact is not major. Especially if I compare it with the third variation of the same code, where I use cudaMalloc instead of managed memory:
Type Time(%) Time Calls Avg Min Max Name
0.00% 164.80us 2 82.401us 82.209us 82.593us [CUDA memcpy HtoD]
0.00% 81.665us 1 81.665us 81.665us 81.665us [CUDA memcpy DtoH]
I am running the NVidia Quadro P1000 laptop, Ubuntu 18.04, Cuda 11.8.
To summarize, here are my questions:
Why does prefetch the memory to the host almost have no impact (29 migrations vs 20 migrations)?
Why is more memory transferred to the host than requested (3Mb instead of the requested 1MB)?
Why even with prefetching the managed memory is the order of magnitude slower than the device memory allocated with cudaMalloc?

Answer 1

As Robert Crovella has mentioned in the comment, the behavior is caused by the fact that the initial location of a managed memory is unspecified.
By default, the devices of compute capability lower than 6.x allocate
managed memory directly on the GPU. However, the devices of compute
capability 6.x and greater do not allocate physical memory when
calling cudaMallocManaged(): in this case physical memory is populated
on first touch and may be resident on the CPU or the GPU.
In my case, the memory was allocated on the GPU. That explains why there were 3MB transferred from the device to the host and the number of transfers themselves. If I remove the initialization loop, the number of HtoD becomes zero (despite the 2 calls to cudaMemPrefetchAsync) and DtoH becomes one.