What's a good way to zero-out cudaMalloc'd data? - cuda

Question

What's a good way to zero-out cudaMalloc'd data? Assume using cudaMemset or cudaMemsetAsync from the CPU results in synchronization issues with other cuda API calls, forcing you to do something else.
Edit 1
In the first picture below, you can see that Thread 1291997440 has issued a cudaMemcpyAsync and takes a while to execute it. For some reason, this cudaMemcpyAsync seems to be blocking the cudaMemsetAsync, as shown in the second picture below. Note that each CPU thread is queuing up these operations in its own stream. Someone of reputation mentioned offsite that using a kernel instead of using a cudaMemsetAsync call could result in clearing the memory sooner -- that's the reason I pursue this question.
Edit 2
At this point, I've improved the code (by reducing the size of the HtoD and DtoH copies) enough to prevent this issue from showing up. The above pictures are from the previous night. If the comments are 100% true, then there must be some other sliver in the profiling report that I didn't notice. In the newer version of the code, there is no observable difference between using cudaMemsetAsync and calling a kernel to clear out the memory.

Answer 1

Here I show the results of zero'ing out ~5.5GB of data with three different approaches. The code was compiled with -O3 and ran on a V100 with 16 GB of memory.
Approach A: cudaMemset
To establish a baseline, I zero out the the data from the CPU, using cudaMemset. This is very fast, but even the cudaMemsetAsync version can be serialized at run time if lots of cudaMemcpys are in flight.
Result: 6 ms
Approach B: memset
Calling memset may invoked the worst of both the CPU and GPU world. memset has this feeling to it that once you exit it, the data is exactly as you indicated it should be. Of course this isn't true in the presence of other kernels, race conditions, etc. But, that's my guess as to why it's so slow.
Result: 241 ms
Approach C: coalesced writes
Writing in a coalesced manner seems to have the best of both the CPU and GPU worlds. It's as fast as the CPU-issued cudaMemset, and it's also clear to any programmer who makes coalesced writes that of course there are race conditions, etc.
Result: 6 ms
Conclusion
If you can't use cudaMemset[Async] from the CPU, then use coalesced writes with 32 threads or more per block.
Program Output
Starting timer for calling cudaMemset from CPU
Stopping timer for calling cudaMemset from CPU took 0.006015s
Starting timer for calling kernel<80,1> that uses memset
Stopping timer for calling kernel<80,1> that uses memset took 0.393921s
Starting timer for calling kernel<80,2> that uses memset
Stopping timer for calling kernel<80,2> that uses memset took 0.300473s
Starting timer for calling kernel<80,4> that uses memset
Stopping timer for calling kernel<80,4> that uses memset took 0.269686s
Starting timer for calling kernel<80,8> that uses memset
Stopping timer for calling kernel<80,8> that uses memset took 0.241374s
Starting timer for calling kernel<80,16> that uses memset
Stopping timer for calling kernel<80,16> that uses memset took 0.645509s
Starting timer for calling kernel<80,32> that uses memset
Stopping timer for calling kernel<80,32> that uses memset took 0.611437s
Starting timer for calling kernel<80,64> that uses memset
Stopping timer for calling kernel<80,64> that uses memset took 0.611276s
Starting timer for calling kernel<80,128> that uses memset
Stopping timer for calling kernel<80,128> that uses memset took 0.459663s
Starting timer for calling kernel<80,256> that uses memset
Stopping timer for calling kernel<80,256> that uses memset took 0.308788s
Starting timer for calling kernel<80,512> that uses memset
Stopping timer for calling kernel<80,512> that uses memset took 0.595893s
Starting timer for calling kernel<80,1024> that uses memset
Stopping timer for calling kernel<80,1024> that uses memset took 2.552866s
Starting timer for calling kernel<80,1> that performs coalesced writes
Stopping timer for calling kernel<80,1> that performs coalesced writes took 0.136967s
Starting timer for calling kernel<80,2> that performs coalesced writes
Stopping timer for calling kernel<80,2> that performs coalesced writes took 0.068426s
Starting timer for calling kernel<80,4> that performs coalesced writes
Stopping timer for calling kernel<80,4> that performs coalesced writes took 0.039974s
Starting timer for calling kernel<80,8> that performs coalesced writes
Stopping timer for calling kernel<80,8> that performs coalesced writes took 0.017121s
Starting timer for calling kernel<80,16> that performs coalesced writes
Stopping timer for calling kernel<80,16> that performs coalesced writes took 0.008586s
Starting timer for calling kernel<80,32> that performs coalesced writes
Stopping timer for calling kernel<80,32> that performs coalesced writes took 0.006139s
Starting timer for calling kernel<80,64> that performs coalesced writes
Stopping timer for calling kernel<80,64> that performs coalesced writes took 0.006075s
Starting timer for calling kernel<80,128> that performs coalesced writes
Stopping timer for calling kernel<80,128> that performs coalesced writes took 0.006093s
Starting timer for calling kernel<80,256> that performs coalesced writes
Stopping timer for calling kernel<80,256> that performs coalesced writes took 0.006479s
Starting timer for calling kernel<80,512> that performs coalesced writes
Stopping timer for calling kernel<80,512> that performs coalesced writes took 0.006972s
Starting timer for calling kernel<80,1024> that performs coalesced writes
Stopping timer for calling kernel<80,1024> that performs coalesced writes took 0.007354s
Test Implementation
memset_timing.cu
#include <iostream>
#include <numeric>
#include <stdlib.h>
#include "timer.h"
static void CheckCudaErrorAux (const char *, unsigned, const char *, cudaError_t);
#define CUDA_CHECK_RETURN(value) CheckCudaErrorAux(__FILE__,__LINE__, #value, value)
#define round_up(x, multiple) (((x + multiple - 1) / multiple) * multiple)
const long COUNT = 80 << 24;
const int GPU_CACHE_LINE_SIZE_IN_BYTES = 32;
const long SIZE_OF_DATA = sizeof(int) * COUNT;
__global__ void clear_scratch_space_kernel(int * data, int blocks, int threads) {
// BOZO: change the code to just error out if we're any of the border cases below
const int idx = blockIdx.x * threads + threadIdx.x;
long size = sizeof(int) * COUNT;
long size_of_typical_chunk = round_up(size / (blocks * threads), GPU_CACHE_LINE_SIZE_IN_BYTES);
// Due to truncation, the threads at the end won't have anything to do. This is a little sloppy but costs us
// hardly anything in performance, so we do the simpler thing.
long this_threads_offset = idx * size_of_typical_chunk;
if (this_threads_offset > SIZE_OF_DATA) {
return;
}
long size_of_this_threads_chunk;
if (this_threads_offset + size_of_typical_chunk >= SIZE_OF_DATA) {
// We are the last thread, so we do a partial write
size_of_this_threads_chunk = SIZE_OF_DATA - this_threads_offset;
} else {
size_of_this_threads_chunk = size_of_typical_chunk;
}
void * starting_address = reinterpret_cast<void *>(reinterpret_cast<char *>(data) + this_threads_offset);
memset((void *) starting_address, 0, size_of_this_threads_chunk);
}
__global__ void clear_scratch_space_with_coalesced_writes_kernel(int * data, int blocks, int threads) {
if (COUNT % (blocks * threads) != 0) {
printf("Adjust the SIZE_OF_DATA so it's divisible by the number of (blocks * threads)\n");
}
const long count_of_ints_in_each_blocks_chunk = COUNT / blocks;
int block = blockIdx.x;
int thread = threadIdx.x;
const long rounds_needed = count_of_ints_in_each_blocks_chunk / threads;
const long this_blocks_starting_offset = block * count_of_ints_in_each_blocks_chunk;
//printf("Clearing %li ints starting at offset %li\n", count_of_ints_in_each_blocks_chunk, this_blocks_starting_offset);
int * this_threads_base_pointer = &data[this_blocks_starting_offset + thread];
for (int round = 0; round < rounds_needed; ++round) {
*this_threads_base_pointer = 0;
this_threads_base_pointer += threads;
}
}
void set_gpu_data_to_ones(int * data_on_gpu) {
cudaMemset(data_on_gpu, 1, SIZE_OF_DATA);
CUDA_CHECK_RETURN(cudaDeviceSynchronize());
}
void check_gpu_data_is_zeroes(int * data_on_gpu, char * data_on_cpu) {
cudaMemcpy(data_on_cpu, data_on_gpu, SIZE_OF_DATA, cudaMemcpyDeviceToHost);
for (long i = 0; i < SIZE_OF_DATA; ++i) {
if (data_on_cpu[i] != 0) {
printf("Failed to zero-out byte offset %i in the data\n", i);
}
}
}
int main(void)
{
const long count = COUNT;
int * data_on_gpu;
char * data_on_cpu = (char *) malloc(SIZE_OF_DATA);
if (data_on_cpu == NULL) {
printf("Failed to allocate data on cpu");
}
CUDA_CHECK_RETURN(cudaMalloc(&data_on_gpu, sizeof(int) * count));
{
Timer memset_timer("calling cudaMemset from CPU");
memset_timer.start();
CUDA_CHECK_RETURN(cudaMemset(data_on_gpu, 0, SIZE_OF_DATA));
CUDA_CHECK_RETURN(cudaDeviceSynchronize());
memset_timer.stop_and_report();
}
for (int threads = 1; threads <= 1024; threads *= 2) {
set_gpu_data_to_ones(data_on_gpu);
char buffer[200];
sprintf(buffer, "calling kernel<80,%i> that uses memset", threads);
Timer memset_timer(buffer);
memset_timer.start();
clear_scratch_space_kernel<<<80, threads>>>(data_on_gpu, 80, threads);
CUDA_CHECK_RETURN(cudaDeviceSynchronize());
memset_timer.stop_and_report();
check_gpu_data_is_zeroes(data_on_gpu, data_on_cpu);
}
for (int threads = 1; threads <= 1024; threads *= 2) {
set_gpu_data_to_ones(data_on_gpu);
char buffer[200];
sprintf(buffer, "calling kernel<80,%i> that performs coalesced writes", threads);
Timer memset_timer(buffer);
memset_timer.start();
clear_scratch_space_with_coalesced_writes_kernel<<<80, threads>>>(data_on_gpu, 80, threads);
CUDA_CHECK_RETURN(cudaDeviceSynchronize());
memset_timer.stop_and_report();
check_gpu_data_is_zeroes(data_on_gpu, data_on_cpu);
}
free(data_on_cpu);
}
/**
* Check the return value of the CUDA runtime API call and exit
* the application if the call has failed.
*/
static void CheckCudaErrorAux (const char *file, unsigned line, const char *statement, cudaError_t err)
{
if (err == cudaSuccess)
return;
std::cerr << statement<<" returned " << cudaGetErrorString(err) << "("<<err<< ") at "<<file<<":"<<line << std::endl;
exit (1);
}
Timer.h
#include <string>
#include <chrono>
class Timer {
public:
Timer(std::string name_, bool allow_output = true);
virtual ~Timer();
void start();
void start_or_restart();
void stop(bool force_no_output = false);
void report(const int count = 0, bool preface_with_spaces = true);
void stop_and_report(const int count = 0);
double duration_in_seconds();
long duration_in_microseconds();
private:
std::string name;
// even though we call report, we still might suppress output since the output is often a type of debugging info
bool allow_output;
std::chrono::time_point<std::chrono::system_clock> start_time;
std::chrono::time_point<std::chrono::system_clock> end_time;
bool started_before = false;
bool currently_rolling = false; // if timer (i.e., the clock) is currently rolling
double duration = -1.0;
};
Timer.cpp
#include <stdexcept>
#include "timer.h"
Timer::Timer(std::string name_, bool allow_output_) {
name = name_;
allow_output = allow_output_;
}
Timer::~Timer() {
}
void Timer::start() {
#ifdef DEBUG
if(started_before) {
printf("Attempting to start same timer twice. Exiting.\n");
throw std::runtime_error("Attempting to start timer that was previously started");
}
#endif
if (allow_output) {
printf("Starting timer for %s\n", name.c_str());
}
start_time = std::chrono::system_clock::now();
currently_rolling = true;
started_before = true;
duration = 0.0;
}
void Timer::start_or_restart() {
if (currently_rolling) {
throw std::runtime_error("Can't start or restart a timer that's already rolling.");
}
if (!started_before && allow_output) {
printf("Starting timer for %s\n", name.c_str());
}
started_before = true;
start_time = std::chrono::system_clock::now();
currently_rolling = true;
if (duration < 0.0) {
duration = 0.0;
}
}
void Timer::stop(bool force_no_output) {
if (!force_no_output) { // Slight style violation: I prefer nested if's over && statements with two && operators
if (allow_output && duration <= 0.0) {
printf("Stopping timer for %s\n", name.c_str());
}
}
end_time = std::chrono::system_clock::now();
std::chrono::duration<double> elapsed_seconds = end_time - start_time;
currently_rolling = false;
duration += elapsed_seconds.count();
}
void Timer::stop_and_report(const int count) {
stop(true);
report(count, false);
}
double Timer::duration_in_seconds() {
return duration;
}
long Timer::duration_in_microseconds() {
return static_cast<long>(duration * 1000000);
}
void Timer::report(const int count, bool preface_with_spaces) {
std::string preface;
if (preface_with_spaces) {
preface = " ";
} else {
preface = "Stopping ";
}
if (allow_output) {
if (!started_before) {
printf("%stimer for %s was never started\n", preface.c_str(), name.c_str());
} else if (count > 0) {
double average = (duration / static_cast<double>(count)) * 1000.0;
printf("%stimer for %s took %fs, %.3lfus each\n", preface.c_str(), name.c_str(), duration, average * 1000.0);
} else {
printf("%stimer for %s took %fs\n", preface.c_str(), name.c_str(), duration);
}
}
}