Why do I have 200 MB GPU usage even when I only created 1 byte data? - cuda

Question

I run the following code in RTX 3060 and RTX 3080 Ti. And by using nvidia-smi, I found the real GPU usage is 105MB and 247MB for RTX 3060 and RTX 3080 Ti separately. Yet I only have 1 byte data in GPU. Why is this? And Why does the basic GPU usage differ?
// compiled with nvcc -O3 show_basic_gpu_usage.cu -o show_basic_gpu_usage
#include <unistd.h>
#include <iostream>
int main(){
int run_count = 100;
int * ddd;
cudaMalloc(&ddd, 1); // 1 byte
for (int i = 0; i < run_count; i++){
sleep(1);
printf("%d\n" , i);
}
}

Answer 1

Running a CUDA program on a GPU requires something like an operating system, not unlike the way running a typical program you might write on the host system CPU also requires an operating system.
In CUDA this GPU operating system is often referred to as the "CUDA runtime" or perhaps the "CUDA driver". The CUDA runtime does all sorts of administration and housekeeping for the GPU, and it requires (both CPU memory and) GPU memory to do that. Some of this requirement is independent of what your code actually does, some of it may vary based on what your code does.
The memory requirement for this "overhead" can vary based on a number of factors:
exact CUDA version and GPU driver version you are using
the GPU type/architecture
the host operating system
the total amount of GPU memory
which kernels and libraries your code links to or loads
whether or not multiple GPUs are visible to the CUDA runtime
(related) whether or not other consumers are using GPU memory, such as a display driver
and probably other factors
Hundreds of megabytes utilization per GPU for this overhead is common. This overhead is in addition to whatever your program may allocate. It's also common to see variation from one GPU type to another. There isn't any way to exactly predict the amount of overhead that is used, because of the variety of influencing factors.