Link .ll files generated by compiling .cu file with clang - cuda

Question

I am compiling the following code using clang with:
clang++ -std=c++11 -emit-llvm -c -S $1 --cuda-gpu-arch=sm_30. This generates vectoradd-cuda-nvptx64-nvidia-cuda-sm_30.ll and vectoradd.ll files. The goal to run some LLVM analysis passes on the kernel, which would possibly instrument it. So i would like to link the post-analysis IR to an executable but i am not sure how. When i try to link the .ll files with llvm-link i am getting the error Linking globals named '_Z9vectoraddPiS_S_i': symbol multiply defined!. I am not really sure how to achieve this, so any help is appreciated.
#define THREADS_PER_BLOCK 512
__global__ void vectoradd(int *A, int *B, int *C, int N) {
int gi = threadIdx.x + blockIdx.x * blockDim.x;
if ( gi < N) {
C[gi] = A[gi] + B[gi];
}
}
int main(int argc, char **argv) {
int N = 10000, *d_A, *d_B, *d_C;
/// allocate host memory
std::vector<int> A(N);
std::vector<int> B(N);
std::vector<int> C(N);
/// allocate device memory
cudaMalloc((void **) &d_A, N * sizeof(int));
cudaMalloc((void **) &d_B, N * sizeof(int));
cudaMalloc((void **) &d_C, N * sizeof(int));
/// populate host data
for ( size_t i = 0; i < N; ++i) {
A[i] = i; B[i] = i;
}
/// copy to device
cudaMemcpy(d_A, &A[0], N * sizeof(int), cudaMemcpyHostToDevice);
cudaMemcpy(d_B, &B[0], N * sizeof(int), cudaMemcpyHostToDevice);
dim3 block(THREADS_PER_BLOCK, 1, 1);
dim3 grid((N + THREADS_PER_BLOCK - 1) / THREADS_PER_BLOCK, 1, 1);
vectoradd<<<grid,block>>>(d_A, d_B, d_C, N);
cudaDeviceSynchronize();
cudaMemcpy(&C[0], d_C, N * sizeof(int), cudaMemcpyDeviceToHost);
return 0;
}

Answer 1

The CUDA compilation trajectory in Clang is rather complicated (as it is in the NVIDIA toolchain) and what you are trying to do cannot work. The LLVM IR from each branch of the compilation process must remain separate until directly linkable objects are available. As a result, there are many intermediate steps which you will need to perform manually.
LLVM IR code for the GPU must be compiled firstly to PTX code, and then assembled to a binary payload which can be linked against host object files.
So in your example, you first do something like:
clang++ -std=c++11 -emit-llvm -c -S test.cu --cuda-gpu-arch=sm_52
which emits two llvm IR files test-cuda-nvptx64-nvidia-cuda-sm_52.ll and test.ll. The GPU code then needs to be compiled to PTX (see more about the nvptx backend here):
llc -mcpu=sm_52 test-cuda-nvptx64-nvidia-cuda-sm_52.ll -o test.ptx
Now the PTX code can be assembled into an ELF file which can later be linked by nvcc (or the host linker with an couple of additional steps) in the normal way:
ptxas --gpu-name=sm_52 test.ptx -o test.ptx.o
fatbinary --cuda -64 --create test.fatbin --image=profile=sm_52,file=test.ptx.o
For the host code you do something like
llc test.ll
clang -m64 -c test.s
to produce assembler output from the LLVM IR and then assemble that to an object file.
Now with a fatbin file containing CUDA the compiled code, and an object file containing the compiled host code, you can perform linkage. I have not been able to test linking a host object file with a fatbinary using clang, that is something you will need to work out yourself. It will be instructive to study both the verbose output of clang during a CUDA compilation call, and also the nvcc documentation to get a better feel for how the device code build system works.