When I go to /usr/local/cuda/samples/1_Utilities/deviceQuery and execute
moose#pc09 /usr/local/cuda/samples/1_Utilities/deviceQuery $ sudo make clean
rm -f deviceQuery deviceQuery.o
rm -rf ../../bin/x86_64/linux/release/deviceQuery
moose#pc09 /usr/local/cuda/samples/1_Utilities/deviceQuery $ sudo make
"/usr/local/cuda-7.0"/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o deviceQuery.o -c deviceQuery.cpp
"/usr/local/cuda-7.0"/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o deviceQuery deviceQuery.o
mkdir -p ../../bin/x86_64/linux/release
cp deviceQuery ../../bin/x86_64/linux/release
moose#pc09 /usr/local/cuda/samples/1_Utilities/deviceQuery $ ./deviceQuery
I keep getting
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version Result = FAIL
I have no idea how to fix it.
My System
moose#pc09 ~ $ cat /etc/issue
Linux Mint 17 Qiana \n \l
moose#pc09 ~ $ uname -a
Linux pc09 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
moose#pc09 ~ $ lspci -v | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX Titan Black] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 1066
Kernel driver in use: nvidia
01:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)
Subsystem: NVIDIA Corporation Device 1066
moose#pc09 ~ $ sudo lshw -c video
*-display
description: VGA compatible controller
product: GK110B [GeForce GTX Titan Black]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci#0000:01:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:96 memory:fa000000-faffffff memory:d0000000-d7ffffff memory:d8000000-d9ffffff ioport:e000(size=128) memory:fb000000-fb07ffff
moose#pc09 ~ $ nvidia-settings -q NvidiaDriverVersion
Attribute 'NvidiaDriverVersion' (pc09:0.0): 331.79
moose#pc09 ~ $ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 331.79 Sun May 18 03:55:59 PDT 2014
GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
moose#pc09 ~ $ lsmod | grep -i nvidia
nvidia_uvm 34855 0
nvidia 10703828 40 nvidia_uvm
drm 303102 5 ttm,drm_kms_helper,nvidia,nouveau
moose#pc09 ~ $ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Mon_Feb_16_22:59:02_CST_2015
Cuda compilation tools, release 7.0, V7.0.27
moose#pc09 ~ $ nvidia-smi
Thu Nov 12 11:23:24 2015
+------------------------------------------------------+
| NVIDIA-SMI 331.79 Driver Version: 331.79 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX TIT... Off | 0000:01:00.0 N/A | N/A |
| 26% 35C N/A N/A / N/A | 132MiB / 6143MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
Update your NVIDIA driver. At the moment you have the driver which only supports CUDA 6 or lower, and you are trying to use the CUDA 7.0 toolkit with it.
I ran into this exact same error message with toolkit 8.0 on ubuntu 1604. I tried reinstalling toolkit, cudnn, etc etc and it didn't help. The solution turned out to be very simple: update to the latest NVIDIA driver. I installed NVIDIA-Linux-x86_64-367.57.run and the error went away.
My cent,
this error may be related to the selected GPU mode (Performance/Power Saving Mode), when you select (with nvidia-settings utiliy) the integrated Intel GPU and you execute the deviceQuery script... you get this error:
-> CUDA driver version is insufficient for CUDA runtime version
But this error is misleading, by selecting back the NVIDIA GPU(Performance mode) with nvidia-settings utility the problem disappears.
It is not a version problem (in my scenario).
Regards
Related
I try to build one of the samples using
make -C 0_Simple/vectorAdd
I get the following output:
/Developer/NVIDIA/CUDA-9.0/bin/nvcc -ccbin clang++ -I…/…/common/inc -m64 -Xcompiler -arch -Xcompiler x86_64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_60,code=compute_60 -o vectorAdd.o -c vectorAdd.cu
nvcc fatal : The version (‘40001’) of the host compiler (‘clang’) is not supported
make: *** [vectorAdd.o] Error 1
I have OSX 10.12.6 (sierra)
I installed XCode 8.3.3 and have set it using sudo xcode-select -s /Applications/Xcode_8.3.3.app/Contents/Developer.
I also installed the command line tools.
This is for CUDA-9.0 since any higher version requires OSX 10.13 or higher.
If I use:
sudo xcode-select --switch /Library/Developer/CommandLineTools/
and then:
make -C 0_Simple/vectorAdd
xcode-select: error: tool 'xcodebuild' requires Xcode, but active
developer directory '/Library/Developer/CommandLineTools' is a command
line tools instance expr: syntax error
/Developer/NVIDIA/CUDA-9.0/bin/nvcc -ccbin g++ -I../../common/inc
-m64 -Xcompiler -arch -Xcompiler x86_64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35
-gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52
-gencode arch=compute_60,code=sm_60 -gencode arch=compute_60,code=compute_60 -o vectorAdd.o -c vectorAdd.cu nvcc
fatal : The version ('90000') of the host compiler ('Apple clang')
is not supported make: *** [vectorAdd.o] Error 1
It is all so unclear, why is it not supported, is it too new? Is it too old?
Should clang be updated or downgraded etc?
In general, CUDA is strictly versioned and requires exact versions of host SDKs and compilers to work. For CUDA 9.0 on OS X, the requirements are
XCode 8.3.3
Clang (Apple LLVM 8.1.0), aka "8100"
(Taken directly from the installation instructions)
If and only if you have those two items correctly installed, CUDA 9.0 will work on Mac OS X 10.12. Otherwise it will not.
I have a docker image of a PyTorch model that returns this error when run inside a google compute engine VM running on debian/Tesla P4 GPU/google deep learning image:
CUDA kernel failed : no kernel image is available for execution on the device
This occurs on the line where my model is called. The PyTorch model includes custom c++ extensions, I'm using this model https://github.com/daveredrum/Pointnet2.ScanNet
My image installs these at runtime
The image runs fine on my local system. Both VM and my system have these versions:
Cuda compilation tools 10.1, V10.1.243
torch 1.4.0
torchvision 0.5.0
The main difference is the GPU as far as I'm aware
Local:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21 Driver Version: 435.21 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 960M Off | 00000000:01:00.0 Off | N/A |
| N/A 36C P8 N/A / N/A | 361MiB / 2004MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
VM:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P4 Off | 00000000:00:04.0 Off | 0 |
| N/A 42C P0 23W / 75W | 0MiB / 7611MiB | 3% Default |
If I ssh into the VM torch.cuda.is_available() returns true
Therefore I suspect it must be something to do with the compilation of the extensions
This is the relevant part of my docker file:
ENV CUDA_HOME "/usr/local/cuda-10.1"
ENV PATH /usr/local/nvidia/bin:/usr/local/cuda-10.1/bin:${PATH}
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=10.1 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419"
ENV FORCE_CUDA=1
# CUDA 10.1-specific steps
RUN conda install -c open3d-admin open3d
RUN conda install -y -c pytorch \
cudatoolkit=10.1 \
"pytorch=1.4.0=py3.6_cuda10.1.243_cudnn7.6.3_0" \
"torchvision=0.5.0=py36_cu101" \
&& conda clean -ya
RUN pip install -r requirements.txt
RUN pip install flask
RUN pip install plyfile
RUN pip install scipy
# Install OpenCV3 Python bindings
RUN sudo apt-get update && sudo apt-get install -y --no-install-recommends \
libgtk2.0-0 \
libcanberra-gtk-module \
libgl1-mesa-glx \
&& sudo rm -rf /var/lib/apt/lists/*
RUN dir
RUN cd pointnet2 && python setup.py install
RUN cd ..
I have already re-running this line from ssh in the VM:
TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0" python setup.py install
Which I think targets the installation to the Tesla P4 compute capability?
Is there some other setting or troubleshooting step I can try?
I didn't know anything about docker/VMs/pytorch extensions until a couple of days ago, so somewhat shooting in the dark. Also this is my first stackoverflow post, apologies if I'm not following some etiquette, feel free to point out.
I resolved this in the end by manually deleting all the folders except for "src" in the folder containing setup.py
Then rebuilt the docker image
Then when building the image I ran TORCH_CUDA_ARCH_LIST="6.1" python setup.py install, to install the cuda extensions targeting the correct compute capability for the GPU on the VM
and it worked!
I guess just running setup.py without deleting the folders previously installed doesn't fully overwrite the extension
System Configuration: Ubuntu 16.04, Nvidia GTX 1060
Cuda Tool Kit: 9.0
I installed Cuda 9.0 on my system and am able to output nvidia-smi
However, when I am trying to make darknet with GPU I am getting the following error:
nvcc -gencode arch=compute_30,code=sm_30 -gencode
arch=compute_35,code=sm_35 -gencode
arch=compute_50,code=[sm_50,compute_50] -gencode
arch=compute_52,code=[sm_52,compute_52] -gencode
arch=compute_61,code=[sm_61,compute_61] -DGPU
-I/usr/local/cuda/include/ --compiler-options "-Wall -Wfatal-errors -Wno-unused-result -Wno-unknown-pragmas -Ofast -DGPU" -c ./src/convolutional_kernels.cu -o obj/convolutional_kernels.o /bin/sh:
nvcc: command not found make: *** [obj/convolutional_kernels.o] Error
127
I had the same error and found out that the cuda path was not correctly added.
These are some post installation steps necessary after CUDA installation.
You might as well add them to your ~/.bashrc file.
$ export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
$ export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Post Installation Steps
its fixed for me thx.
export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
i am using cuda 10.1 version and this also working for that version.
It also worked for me thanks.
export PATH=/usr/local/cuda-11.5/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.5/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
I am using cuda 11.5 version so I just changed it.
When I go to /usr/local/cuda/samples/1_Utilities/deviceQuery and execute
moose#pc09 /usr/local/cuda/samples/1_Utilities/deviceQuery $ sudo make clean
rm -f deviceQuery deviceQuery.o
rm -rf ../../bin/x86_64/linux/release/deviceQuery
moose#pc09 /usr/local/cuda/samples/1_Utilities/deviceQuery $ sudo make
"/usr/local/cuda-7.0"/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o deviceQuery.o -c deviceQuery.cpp
"/usr/local/cuda-7.0"/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o deviceQuery deviceQuery.o
mkdir -p ../../bin/x86_64/linux/release
cp deviceQuery ../../bin/x86_64/linux/release
moose#pc09 /usr/local/cuda/samples/1_Utilities/deviceQuery $ ./deviceQuery
I keep getting
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version Result = FAIL
I have no idea how to fix it.
My System
moose#pc09 ~ $ cat /etc/issue
Linux Mint 17 Qiana \n \l
moose#pc09 ~ $ uname -a
Linux pc09 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
moose#pc09 ~ $ lspci -v | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX Titan Black] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 1066
Kernel driver in use: nvidia
01:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)
Subsystem: NVIDIA Corporation Device 1066
moose#pc09 ~ $ sudo lshw -c video
*-display
description: VGA compatible controller
product: GK110B [GeForce GTX Titan Black]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci#0000:01:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:96 memory:fa000000-faffffff memory:d0000000-d7ffffff memory:d8000000-d9ffffff ioport:e000(size=128) memory:fb000000-fb07ffff
moose#pc09 ~ $ nvidia-settings -q NvidiaDriverVersion
Attribute 'NvidiaDriverVersion' (pc09:0.0): 331.79
moose#pc09 ~ $ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 331.79 Sun May 18 03:55:59 PDT 2014
GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
moose#pc09 ~ $ lsmod | grep -i nvidia
nvidia_uvm 34855 0
nvidia 10703828 40 nvidia_uvm
drm 303102 5 ttm,drm_kms_helper,nvidia,nouveau
moose#pc09 ~ $ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Mon_Feb_16_22:59:02_CST_2015
Cuda compilation tools, release 7.0, V7.0.27
moose#pc09 ~ $ nvidia-smi
Thu Nov 12 11:23:24 2015
+------------------------------------------------------+
| NVIDIA-SMI 331.79 Driver Version: 331.79 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX TIT... Off | 0000:01:00.0 N/A | N/A |
| 26% 35C N/A N/A / N/A | 132MiB / 6143MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
Update your NVIDIA driver. At the moment you have the driver which only supports CUDA 6 or lower, and you are trying to use the CUDA 7.0 toolkit with it.
I ran into this exact same error message with toolkit 8.0 on ubuntu 1604. I tried reinstalling toolkit, cudnn, etc etc and it didn't help. The solution turned out to be very simple: update to the latest NVIDIA driver. I installed NVIDIA-Linux-x86_64-367.57.run and the error went away.
My cent,
this error may be related to the selected GPU mode (Performance/Power Saving Mode), when you select (with nvidia-settings utiliy) the integrated Intel GPU and you execute the deviceQuery script... you get this error:
-> CUDA driver version is insufficient for CUDA runtime version
But this error is misleading, by selecting back the NVIDIA GPU(Performance mode) with nvidia-settings utility the problem disappears.
It is not a version problem (in my scenario).
Regards
I am trying to run the cuda's printf example that I found in this thread:
#include <stdio.h>
__global__ void helloCUDA(float f)
{
printf("Hello thread %d, f=%f\n", threadIdx.x, f);
}
int main()
{
helloCUDA<<<1, 5>>>(1.2345f);
cudaDeviceReset();
return 0;
}
It's being compiled with:
nvcc -arch=sm_20 test.cu -run
I got no output as well:
$ nvcc -arch=sm_20 test.cu -run
$
Here is my cuda version:
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2012 NVIDIA Corporation
Built on Fri_Sep_21_17:28:58_PDT_2012
Cuda compilation tools, release 5.0, V0.2.1221
I am using bumblebee version 3:
$ optirun --version
optirun (Bumblebee) 3.1
Copyright (C) 2011 The Bumblebee Project
$ uname -a
Linux zeus 3.5.0-25-generic #39-Ubuntu SMP Mon Feb 25 18:26:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
change that cudaDeviceReset() call to a cudaDeviceSynchronize() call. Also do cuda error checking on the cudaDeviceSynchronize() call. There may be a problem with your GPU or setup.