Segmentation fault when compiling Darknet for GPU - cuda

I want to compile the Darknet framework for machine learning on my PC with GPU support. However I call make I will get a segmentation fault:
nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Iinclude/ -Isrc/ -DOPENCV `pkg-config --cflags opencv` -DGPU -I/usr/local/cuda/include/ --compiler-options "-Wall -Wno-unused-result -Wno-unknown-pragmas -Wfatal-errors -fPIC -Ofast -DOPENCV -DGPU" -c ./src/convolutional_kernels.cu -o obj/convolutional_kernels.o
Segmentation fault (core dumped)
Makefile:92: recipe for target 'obj/convolutional_kernels.o' failed
make: *** [obj/convolutional_kernels.o] Error 139
nvidia-smi gives me following information:
NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1
When I do nvcc --version I get:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85
The CUDA Version 10.1 is not the same as the Verions 9.1 of the Cuda compilation tools. Could this be the problem? NVCC is installed via apt install nvidia-cuda-toolkit

Just gonna post my solution here because I figured out the actual reason for this. So the reason this happens is because it's running a different binary than the actual one darknet wants to run. At least for me, which nvcc gave me /usr/bin/nvcc. The actual nvcc you want is located in /usr/local/cuda-11.1/bin (version number might be different obviously). So all you need to do is prepend (important!) that directory to your PATH variable.
export PATH=/usr/local/cuda-11.1/bin${PATH:+:${PATH}} >> ~/.bashrc
Source:https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
I recommend you follow the link because there are a couple more mandatory post-installation steps that I also did not follow.

I solved the problem. After installing cuda the actual binary of nvcc is at /usr/local/cuda/bin/nvcc. Creating a symbolic link in /usr/bin/ to this binary solved the problem.

Another approach is to edit the Makefile and set the correct nvcc.
In my case:
line 24 replace
NVCC=nvcc
to
NVCC=/usr/local/cuda-11.0/bin/nvcc
Note that the cuda version may vary.

Related

Undefined reference to `cublasCreate_v2’ in ‘/tmp/tmpxft_0000120b_0000000-10_my_program”

I have tried to compile a code using CUDA 9.0 toolkit on NVIDIA Tesla P100 graphic card (Ubuntu version 16.04) and CUBLAS library is used in the code. For compilation, I have used the following command to compile “my_program.cu”
nvcc -std=c++11 -L/usr/local/cuda-9.0/lib64 my_program.cu -o mu_program.o -lcublas
But, I have got the following error:
nvlink error: Undefined reference to 'cublasCreate_v2’in '/tmp/tmpxft_0000120b_0000000-10_my_program’
As I have already linked the library path in the compilation command, why do I still get the error. Please help me to solve this error.
It seems fairly evident that you are trying to use the CUBLAS library in device code. This is different than ordinary host usage and requires special compilation/linking steps. You need to:
compile for the correct device architecture (must be cc3.5 or higher)
use relocatable device code linking
link in the cublas device library (in addition to the cublas host library)
link in the CUDA device runtime library
Use a CUDA toolkit prior to CUDA 10.0
The following additions to your compile command line should get you there:
nvcc -std=c++11 my_program.cu -o my_program.o -lcublas -arch=sm_60 -rdc=true -lcublas_device -lcudadevrt
The above assumes you are actually using a proper install of CUDA 9.0. The CUBLAS device library was deprecated and is now removed from newer CUDA toolkits (see here).

Compile error during Caffe installation on OS X 10.11

I've configured Caffe environment on my Mac for several times. But this time I encountered a problem I've never met before:
I use Intel's MKL for accelerating computation instead of ATLAS, and I use Anaconda 2.7 and OpenCV 2.4, with Xcode 7.3.1 on OS X 10.11.6.
when I
make all -j8
in terminal under Caffe's root directory, the error info is:
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0-rc5
clang: warning: argument unused during compilation: '-pthread'
ld: can't map file, errno=22 file '/usr/local/cuda/lib' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc5] Error 1
make: *** Waiting for unfinished jobs....
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(parallel.o) has no symbols
I've tried many times, does anyone can help me out?
This looks like you haven't changed Makefile.config from GPU to CPU mode. There shouldn't be anything trying to actively link that library. I think the only CUDA one you should need is libicudata.so
Look for the lines
# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1
and remove the octothorpe from the front of the second line.

Trouble compiling/running CUDA code involving dynamic parallelism

I am trying to use dynamic parallelism with CUDA, but I cannot go through the compilation step.
I am working on a GPU with Compute Capability 3.5 and the CUDA version 7.5.
Depending on the switches in the compile command I use, I am getting different error messages, but using the documentation,
I arrived to one line leading to a successful compilation:
nvcc -arch=compute_35 -rdc=true cudaDynamic.cu -o cudaDynamic.out -lcudadevrt
But when the program is launched, all the program fails. With
CUDA-memcheck, for each call to an API function, I get the same error
message:
========= CUDA-MEMCHECK
========= Program hit cudaErrorUnknown (error 30) due to "unknown error" on CUDA API call to ...
I have also tried this line (taken from CUDA dynamic samples makefile):
nvcc -ccbin g++ -I../../common/inc -m64 -dc -gencode arch=compute_35,code=compute_35 -o cudaDynamic.out -c cudaDynamic.cu
But upon execution, I get:
cudaDynamic.out: Permission denied
I would like to understand how to correctly compile a CUDA dynamic code, because all the other compilation lines that I have tried so far have failed.
I fixed the problem by fully reinstalling CUDA.
I'm now able to compile both the CUDA samples and my own code.

Why is nvlink warning me about lack of sm_20 (compute capability 2.0) object code?

I'm working with CUDA 6.5 on a machine with a GTX Titan card (compute capability 3.5). I'm building my code with just -gencode=arch=compute_30,code=sm_30 -gencode=arch=compute_35,code=sm_35 - and when I link my binary, nvlink says:
nvlink warning : SM Arch ('sm_20') not found in '/local/eyalroz/src/foo/CMakeFiles/tester.dir/src/./tester_generated_main.cu.o'
Why is it warning me about that? Do I need sm_20 for something I'm not aware of? If it's merely about the lack of lower compute capability support, why not sm_10 as well? (Also, how do I turn off the warning, if it's gratuitous?)
The issue was identified in CUDA 6.5 and has been rectified I believe in CUDA 7.5. Using the latest version of CUDA should make those warnings go away.
Just ignore it
i'm on cuda RC8 and have the same issue:
nvlink warning : SM Arch ('sm_20') not found in 'cudainfo.o'
compile:
/usr/local/cuda/bin/nvcc -g -O2 -Iyes/include -Iyes/include -I. -gencode arch=compute_35,code=sm_35 -rdc=true --ptxas-options=-v -I./compat/jansson -o cudainfo.o -c cudainfo.cu
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Wed_May__4_21:01:56_CDT_2016
Cuda compilation tools, release 8.0, V8.0.26

no cuda compatible device detected on nsight eclipse. why?

i'm writing a simple code for fast fourier transform with cufft cuda library. My source file work well with visual studio in windows7 but with eclipse nsight, in ubuntu 14.04, not work!
i've installed nvidia 346.72 driver and cuda toolkit 7.0 and my video hardware is geforce 410M. When i build my source code i have following message:
16:56:24 **** Incremental Build of configuration Debug for project cufft_double ****
make all
Building target: cufft_double
Invoking: NVCC Linker
/usr/local/cuda-7.0/bin/nvcc --cudart static -L/usr/local/cuda-7.0/lib64 --relocatable-device-code=false -gencode arch=compute_20,code=compute_20 -gencode arch=compute_20,code=sm_20 -m64 -link -o "cufft_double" ./cufft_double.o
./cufft_double.o: In function `main':
/home/marco/cuda-workspace/cufft_double/Debug/../cufft_double.cu:79: undefined reference to `cufftPlan1d'
/home/marco/cuda-workspace/cufft_double/Debug/../cufft_double.cu:85: undefined reference to `cufftExecZ2Z'
/home/marco/cuda-workspace/cufft_double/Debug/../cufft_double.cu:108: undefined reference to `cufftDestroy'
/home/marco/cuda-workspace/cufft_double/Debug/../cufft_double.cu:111: undefined reference to `cufftPlan1d'
/home/marco/cuda-workspace/cufft_double/Debug/../cufft_double.cu:117: undefined reference to `cufftExecZ2Z'
/home/marco/cuda-workspace/cufft_double/Debug/../cufft_double.cu:136: undefined reference to `cufftDestroy'
collect2: error: ld returned 1 exit status
make: *** [cufft_double] Error 1
16:56:27 Build Finished (took 2s.792ms)
i tried to set library path but in preferences windows i read "no CUDA-compatible devices detected"
please help me!
Best reguards
marco
now i can build source code but my program not work!
i read this error:
modprobe: ERROR: could not insert 'nvidia_331_uvm': Invalid argument
and i receive a message programmed by me if "cudaGetLastError() != cudaSuccess"
after "cudaMalloc"
For best clarification i read "cuda error: allocazione fallita" for this frame of code:
cudaMalloc((void**)&out_device, sizeof(cufftDoubleComplex)*NX*BATCH);
if (cudaGetLastError() != cudaSuccess){
printf("Cuda error: allocazione fallita\n");
return 0;
};
Run these commands in sequence:
sudo apt-get remove --purge nvidia-*
sudo apt-get install cuda-drivers
sudo apt-get install nvidia-nsight
Restart the machine and open nsight and look at the properties whether you see the detected driver.