Why am I getting this cuda out of memory error on my new computer and not on the old one? - cuda

I'm currently trying to extract SIFT Features with the following package:
https://github.com/Celebrandil/CudaSift
It comes with a CMakeLists.txt, which I modified, here it is:
cmake_minimum_required(VERSION 2.6)
project(cudaSift)
set(cudaSift_VERSION_MAJOR 2)
set(cudaSift_VERSION_MINOR 0)
set(cudaSift_VERSION_PATCH 0)
set(CPACK_PACKAGE_VERSION_MAJOR "${cudaSift_VERSION_MAJOR}")
set(CPACK_PACKAGE_VERSION_MINOR "${cudaSift_VERSION_MINOR}")
set(CPACK_PACKAGE_VERSION_PATCH "${cudaSift_VERSION_PATCH}")
set(CPACK_GENERATOR "ZIP")
include(CPack)
find_package(OpenCV REQUIRED)
find_package(CUDA)
if (NOT CUDA_FOUND)
message(STATUS "CUDA not found. Project will not be built.")
endif(NOT CUDA_FOUND)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2 -msse2 ")
list(APPEND CUDA_NVCC_FLAGS "-lineinfo;-ccbin;/usr/bin/gcc-7;--compiler-options;-O2;-D_FORCE_INLINES;-DVERBOSE_NOT; -arch=sm_75")
cuda_add_library(cudaSift SHARED
src/cudaImage.cu
src/cudaSiftH.cu
src/matching.cu
src/geomFuncs.cpp
src/mainSift.cpp
)
target_link_libraries(cudaSift ${CUDA_cudadevrt_LIBRARY} ${OpenCV_LIBS})
set(PUBLIC_HEADERS include/cudaImage.h include/cudaSift.h)
set_target_properties(cudaSift PROPERTIES PUBLIC_HEADER
"${PUBLIC_HEADERS}"
)
include(GNUInstallDirs)
install(TARGETS cudaSift
LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
)
configure_file(cudaSift.pc.in cudaSift.pc #ONLY)
install(FILES ${CMAKE_BINARY_DIR}/cudaSift.pc DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/pkgconfig)
My GPU is a GeForce RTX 2060, driver version 430.5, and after running:
mkdir build && cd build
cmake ..
sudo make -j
sudo make install
sudo ldconfig
-in order to build the package, I try to run my code and get the following error:
safeCall() Runtime API error in file </path/to/CudaSift/src/cudaImage.cu>, line 24 : out of memory.
Precisions:
I run the exact same code on another computer which has a GeForce GTX 1050, only changing in CMakeLists.txt -arch=sm_75 to arch=sm_61 and it executes just fine.
From previous questions, I think this is a compilation problem, linked to the arch=sm_** value, but I changed it and it still doesn't work.
The objects i'm passing to my GPU are images, which I'm sure aren't too big, since it works on my other computer which GPU has less memory

UPDATE:
I found the problem, the package was actually compiled properly.
Actually A tensorflow model was loaded in the code, but after deleting it, the error didn't happen again.
I don't know why though, maybe it reserved a lot of GPU memory ?

Related

ERR_NVGPUCTRPERM error when launching nvprof with all metrics to profile CUDA application

GPU Tesla M60
Driver: 510.47.03
OSL Ubuntu 20.04.5 LTS
CUDA Version: 11.6
Trying the code below to get back full metrics on profiling a CUDA application results in teh error below.
Code
nvprof --metrics all ./myapp
Error
==8169== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM
I tried using sudo as suggested but was unable to find the nvcc program.
The easiest solution is to run the profiler as root as below, noting that it may be necessary to use the fully qualified path to find nvcc if it is not in your sudo path.
sudo /usr/local/cuda/bin/nvprof --metrics all ./myapp
There are more permanent solutions available as per https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters such as changing permission settings with modprobe. However, I was not able to get these to work.

CUDA: safeCall() Runtime API error invalid device symbol [duplicate]

I use the cmake gui tool to configure my cuda project in vs2013.
CMakeLists.txt is as below:
project(CUDA_PART)
# required cmake version
cmake_minimum_required(VERSION 3.0)
include_directories(${CUDA_PART_SOURCE_DIR}/common)
# packages
find_package(CUDA REQUIRED)
# nvcc flags
set(CUDA_NVCC_FLAGS -gencode arch=compute_20,code=sm_20;-G;-g)
set(CUDA_VERBOSE_BUILD ON)
#FILE(GLOB SOURCES "*.cu" "*.cpp" "*.c" "*.h")
CUDA_ADD_EXECUTABLE(CUDA_PART hist_gpu_shmem_atomics.cu)
The .cu file is from Cuda by example source code hist_gpu_shmem_atomics.cu
There are two problems:
After the line histo_kernel <<<blocks * 2, 256 >>>(dev_buffer, SIZE, dev_histo);an "invalid device function" error occurs.
When I use the CUDA debugging tool to debug, its cannot trigger breakpoints in the device code.
But when I create a project with the same code by the cuda project temple in visual studio 2013.It works correctly!
So, is there something wrong in the CMakeLists.txt ?
OS: Win7 64bit;GPU: GTX960;CUDA: CUDA 7.5;VS: 2013 (and also 2010)
When I use set the "Code Generation" in vs2013 as follow :
The CUDA_NVCC_FLAGES turns out to be -gencode=arch=compute_20,code=\"sm_20,compute_20\"
It equals to:
-gencode=arch=compute_20,code=sm_20 \
-gencode=arch=compute_20,code=compute_20
So, I guess it will generate 2 versions machine code: the first one(SASS) with virtual and real architectures and the second one(PTX) with only virtual architecture. Since my GTX960 is a cc5.2 device, it chooses the second one (PTX) and convert it to a suitable SASS.
This is a problem:
set(CUDA_NVCC_FLAGS -gencode arch=compute_20,code=sm_20;-G;-g)
Those flags will cause nvcc to generate SASS code (only) for a cc 2.0 device (only). Such cc2.0 SASS code will not run on your cc5.2 device (GTX960). "Invalid device function" is exactly the error you would get when trying to launch a kernel in such a scenario. Since the kernel will never launch, trying to hit breakpoints in device code won't work.
I'm not a CMake expert, so there might be other, more sensible approaches, but one possible way to try to fix this might be:
set(CUDA_NVCC_FLAGS -gencode arch=compute_52,code=sm_52;-G;-g)
which should generate code for your cc5.2 device. There are undoubtedly other possible settings here, you may want to read this or the nvcc manual for more background on compile options to target specific devices.
Also note that -G generates device debug code, which is fine if that is what you want. However it will generally run slower than code compiled without that switch. If you want to debug, however, that switch is necessary.

CUDA 7.0 Error while compiling samples

I'm trying to install CUDA 7.0 on Ubuntu 14.04. I've followed the installation instructions as outlined here. Specifically, I've followed steps in section 3.6 and Chapter 6. While compiling the examples (Section 6.2.2.2) using make, I'm getting the following error:
make[1]: Entering directory `/usr/local/cuda-7.0/samples/3_Imaging/cudaDecodeGL'
/usr/local/cuda-7.0/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_20,
code=compute_20 -o cudaDecodeGL FrameQueue.o ImageGL.o VideoDecoder.o
VideoParser.o VideoSource.o cudaModuleMgr.o cudaProcessFrame.o
videoDecodeGL.o -L../../common/lib/linux/x86_64 -L/usr/lib/"nvidia-346"
-lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW -lcuda -lcudart -lnvcuvid
/usr/bin/ld: cannot find -lnvcuvid
collect2: error: ld returned 1 exit status
make[1]: *** [cudaDecodeGL] Error 1
make[1]: Leaving directory `/usr/local/cuda-7.0/samples/3_Imaging/cudaDecodeGL'
make: *** [3_Imaging/cudaDecodeGL/Makefile.ph_build] Error 2
If you notice, there is -L/usr/lib/"nvidia-346". In my case, I have installed nvidia-349. What worked for me is to edit NVIDIA_CUDA-7.0_Samples/3_Imaging/cudaDecodeGL/findgllib.mk and change UBUNTU_PKG_NAME = "nvidia-346" to nvidia-349.
In order to properly install CUDA 7.0 on Ubuntu 14.04, you need a nvidia driver version 346 or higher.
If you're using the .deb installation method, the nvidia graphics driver is installed automatically.
If you used the .run file installation method and chose not to install the nvidia driver, you can manually install the driver afterwards through the package manager:
sudo apt-add-repository ppa:xorg-edgers/ppa && sudo apt-get update
sudo apt-get install nvidia-346 nvidia-346-dev nvidia-346-uvm libcuda1-346 nvidia-libopencl1-346 nvidia-icd-346
In my case, I installed nvidia-352 afterwards due to a bug in nvidia-346 and I stumbled upon the same error.
andoum's approach of manually changing the hard-coded UBUNTU_PKG_NAME = "nvidia-346" to UBUNTU_PKG_NAME = "nvidia-352" in NVIDIA_CUDA-7.0_Samples/3_Imaging/cudaDecodeGL/findgllib.mk worked fine for me.
I met the same issue and solution is that put path of nvidia into system path:
sudo gedit /etc/environment
add these path into environment
LIBRARY_PATH=/usr/lib/your_nvidia_edition:$LIBRARY_PATH
In fact I have encountered this problem when I made a make. I installed Cuda 8.0 under my Ubuntu 16.04. This problem had been confusing me for several weeks and I was almost tending to reinstall ubuntu for that after reviewing many suggestions via google, but finally I addressed it myself recently.
First of all, you should replace all the UBUNTU_PKG_NAME= ##nvidia-3xx## to the one of your actually installed nvidia driver version as recommended above. Then you will probably get compiling error after you do a new make. In my case, I have the link errors like
/usr/bin/ld: warning: libGLX.so.0, needed by /usr/lib/nvidia-
375/libGL.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libGLdispatch.so.0, needed by /usr/lib/nvidia-
375/libGL.so, not found (try using -rpath or -rpath-link)
....
or whatever contains missing link errors. Do locate the files you miss like
$ locate libGLX.so.
/usr/lib/nvidia-375/libGLX.so.0
/usr/lib32/nvidia-375/libGLX.so.0
$ locate libGLdispatch.so.0
/usr/lib/nvidia-375/libGLdispatch.so.0
/usr/lib32/nvidia-375/libGLdispatch.so.0
The error above is probably caused the compiling files cannot find in the default cuda libraries as you set, so you just need to copy the missing files to /usr/lib/nvidia-3xx/ (the actual path in your case) and this should work(it works in my case), if it doesn't maybe you could try to link the new add files to the one that need using a
$ sudo ln -s (requested file) (requesting file).
Hope this will help.

MIPS GCC compiling wrong MIPS release?

I'm looking to plunge into the realm of router hacking. Currently, I'm just trying to get a simple hello world to run on my router, a F7D7302 v1, running dd-wrt. cpuinfo reveals a BCM4716, and "MIPS 74K V4.0" as the cpu 'model.'
I downloaded CodeSourcery's MIPS cross-compiler. It was pretty simple to get a compiled executable from that point on. I transferred the file, chmod +x'd the file, and it didn't run.
I decided to do a big of snooping and ran 'file' on the router's busybox executable:
[root#LIENUX bin]# file /tmp/busybox
/tmp/busybox: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), dynamically linked (uses shared libs), corrupted section header size
And running 'file' on my hello world:
[root#LIENUX bin]# file test
test: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1, dynamically linked (uses shared libs), for GNU/Linux 2.6.16, with unknown capability 0xf41 = 0x756e6700, with unknown capability 0x70100 = 0x1040000, not stripped
Running test on the router with static libraries caused a 100% CPU hang; running with dynamic libraries caused a 'not found' error. All appropriate permissions were set.
Anyone know what I am doing wrong?

Win7 MInGW QT MySQL program screams "cannot find -lqsqlmysqld"; where is the missing library?

I am attempting to make a qt program on Windows 7 that uses a MySQL plugin.
I have compiled both qt and the mysql plugin with no problems using my minGW 32bit compiler.
However, I keep on getting an error like this:
mingw32-make -f Makefile.Debug
mingw32-make[1]: Entering directory `C:/Users/dhatt/Desktop/testdb2'
g++ -c -g -frtti -fexceptions -mthreads -Wall -DUNICODE -DQT_LARGEFILE_SUPPORT -DQT_DLL -DQT_SQL_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_THREAD_SUPPORT -DQT_NEEDS_QMAIN - I"..\..\..\..\QT\qt\include\QtCore" -I"..\..\..\..\QT\qt\include\QtGui" -I"..\..\..\..\QT\qt\include\QtSql" -I"..\..\..\..\QT\qt\include" -I"..\..\..\..\MySQL\bin" -I"..\..\..\..\QT\qt\include\ActiveQt" -I"debug" -I"..\..\..\..\QT\qt\mkspecs\win32-g++" -o debug\database.o database.cpp
g++ -enable-stdcall-fixup -Wl,-enable-auto-import -Wl,-enable-runtime-pseudo-reloc -mthreads -Wl -Wl,-subsystem,windows -o debug\testdb2.exe debug/database.o -L"c:\QT\qt\lib" -lmingw32 -lqtmaind -L C:\MySQL\lib\opt -LC:/QT/qt/plugins/sqldrivers -lqsqlmysqld -lQtSqld4 -lQtGuid4 -lQtCored4 -LC:\MySQL\lib\opt
C:/qt/mingw/bin/../lib/gcc/mingw32/4.4.0/../../../../mingw32/bin/ld.exe: cannot find -lqsqlmysqld
collect2: ld returned 1 exit status
mingw32-make[1]: *** [debug\executable.exe] Error 1
mingw32-make[1]: Leaving directory `C:/Users/dhatt/Desktop/testdb2'
mingw32-make: *** [debug] Error 2
I apologize in advance for being very verbose of what I did, but I am doing this partly for troubleshooting, and partly so any other lost souls don't end up wasting three weeks on this particular problem. :)
Here are my specs:
Windows 7 Nokia's Open Source QT
Qt SDK for Windows (C:\Qt\2010.04\qt)
Linux MinGW Version 5.1.6 (C\Linux\MinGW)
MySQL5 with C++ files (C:\MySQL5)
If you want to know how I installed qt, just follow this site's instructions:
http://www.jiggerjuice.net/software/qt-sql-drivers.html
These other sites may hold some extra information tidbits:
http://doc.qt.nokia.com/4.6/sql-driver.html
http://www.rag.com.au/linux/qt4howto.html
http://qtnode.net/wiki?title=Qt4_on_Windows (yes, I did check with Nokia's docs!!!)
http://doc.trolltech.com/qq/qq10-windows-deployment.html
This fellow mentioned about remaking qmake, which I am not doing unless I have a good reason.
http://christopher.rasch-olsen.no/2009/04/14/qt-45-and-mysql-plugin-with-mingw-on-windows-xp/
I've already deleted the plugin cache once before, I hope I won't have to do it again...
http://doc.trolltech.com/4.2/plugins-howto.html#the-plugin-cache
http://ubuntuforums.org/showthread.php?t=1070155
If there is any confusion, between the two compilation option (creating the mysql libraries statically, or as a plugin), I chose for the plugin because it compiles quicker and I don't have to worry about licensing.
Generally, the big trouble of mysql to most people is to make a mingw compatible library. Generally, I did this with the mingw tools in ( https://olex.openlogic.com/packages/mingw-utils )...
c:\> cd MySQL\lib\opt
c:\mysql\lib\opt> reimp -d libmysql.lib
c:\mysql\lib\opt> dlltool --input-def libmysql.def --dllname libmysql.dll --output-lib libmysql.a -k
I should have done it right since in my C:\MySQL\lib\opt, it has the two files:
libmysql.a
libmysql.lib
LIBMYSQL.def (not a typo)
and in the C:\MySQL\bin directory, I have:
libmySQL.bin (not a typo)
I had compiled the mysql plugin beforehand:
cd %QTDIR%\src\plugins\sqldrivers\mysql
qmake "INCLUDEPATH+=C:\MySQL\include" "LIBS+=C:\MYSQL\lib\opt\libmysql.lib" mysql.pro
mingw32-make
As a result, I have in my C:\QT\qt\plugins\sqldrivers folder:
libqsqlmysql4.a
libqsqlmysqldq4.a
libqsqlodbc4.a
libqsqlodbcd4.a
qsqlmysql4.dll
qsqlmysqld4.dll
qsqlodbc4.dll
qsqlodbc4.dll
And in my C:\QT\bin folder
QtSql4.dll
QtSqld4.dll
So, I assume from this site ( http://www.qtforum.org/article/21352/how-to-compile-use-a-mysql-driver.html) that I got it right.
I didn't use the binaries of qt itself, I used the compiled qt files(also from Nokia), but reconfigured and recompiled them using mingw32-make. I had no errors. This was my configuration options for remaking qt.
-opensource
-nomake examples
-nomake demos
-no-sql-lite
-no-qt3support
-no-gif
-no-libpng
-no-libmng
-no-libtiff
-no-phonon
-no-phonon-backend
-no-multimedia
-no-audio-backend
-no-webkit
-no-script
-no-scripttools
-nodeclarative
-plugin-sql-mysql -l mysql -I C:\QT\qt\include -L C:\QT\qt\lib\opt
Here is my .pro file
LANGUAGE = C++
TEMPLATE = app
TARGET = executable
QT += core sql
QTPLUGIN += qsqlmysql
DEPENDPATH += .
INCLUDEPATH += C:\MySQL\bin
LIBS += -L C:\MySQL\lib\opt -lmysql
# Input
SOURCES += database.cpp
I installed the plugin described in here:
C:\QT\qt
My path variables are:
%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Python26;C:\Linux\Cygwin\bin;C:\mingw-utils-0.3\bin;C:\QT\qt\bin;C:\MySQL\bin;C:\MySQL\include;C:\QT\mingw32\bin;C:\QT\mingw\bin;C:\QT\qt\plugins\sqldrivers
The qt command prompt added a few extra though, so I did all of this in the command prompt.
Setting up a MinGW/Qt only environment...
-- QTDIR set to C:\QT\qt
-- PATH set to C:\QT\qt\bin
-- Adding C:\QT\bin to PATH
-- Adding C:\Windows\System32 to PATH
-- QMAKESPEC set to win32-g++ (mingw is my only compiler so, this is unnecessary)
Although I either did all that already, or it is redundant. I only add this for the sake of completeness.
Here is my code (database.cpp):
#include <QtSql>
#include <iostream>
using namespace std;
int main( int argc, char ** argv )
{
QSqlDatabase db = QSqlDatabase::addDatabase("QMYSQL");
db.setHostName("---.---.---.---");
db.setDatabaseName("--------");
db.setUserName("------------");
db.setPassword("------------");
if (!db.open()) cout << "Failed to connect to mysql" << endl;
else cout << "Works finally." << endl;
QSqlDatabase::removeDatabase("QMYSQL");
exit ( 0 );
}
Very simple, yes?
I went to my directory with the example code, run
mingw32-make distclean
qmake
mingw32-make
and get the error message above. I've tried building a version with the release version only (no debug), and it still shows the same message, but with "cannot find -lqsqlmysq", so it is not that.
I've tried many things, but where should I look next to solve it; maybe someone can narrow it down for me, set me on the right path, or even better, solve his annoying problem.
Also, I plan to use python bindings with my code (I need PyQT + MySQL). If the proposed solution would prevent me from doing so, let me know.
Well, I'm going to solve my own problem, again, so let's make this fun!
This is your last chance.
After this, there is no turning back.
You download the PyQT.exe, the story ends. You wake up in your bed and you believe whatever you want to believe. You modify the .pro file, you stay in wonderland. And, I show you how deep the rabbit hole goes.
I eventually gave up and downloaded the .exe, which does have MySQL support out of the box. If mysql does not work, your application is the problem, and I recommend you read this post here ( http://lists.trolltech.com/qt-interest/2006-06/thread00292-0.html ) or follow the quote below:
The issue is that you either have to
use the addLibraryPath method or
create a QCoreApplication instance
before your first call to loading a
database
Believe me, manually installing PyQT+MySQL on Windows is a pain. But if you need some out of the way plugin to get at that the executable doesn't know, you have to go down the rabbit hole further.
Here is the new and improved .pro file:
LANGUAGE = C++
TEMPLATE = app
TARGET = executable
QT += core sql
QTPLUGIN += qsqlmysql
DEPENDPATH += .
INCLUDEPATH += C:\MySQL\bin
LIBS += -L C:\MySQL\lib\opt
# Input
SOURCES += database.cpp
Turns out I did have the right path to mysql, I was just confusing it with the .pro file that I had. After redownloading qt and following the steps above again, modifying my .pro file made all the difference.
But now I had to download SIP and PyQT. I followed the docs on there. There are a few more problems. Follow the links or the directions which are left there in case the information is removed.
If your SIP make install has an error where it is looking at Unix paths (/usr/bin) instead of DOS paths (C:\QT), look at this link http://old.nabble.com/Building-SIP-on-MinGW-:-problem-at-%22make-install%22-td28909249.html#
(short version: the problem is the sh.exe in one of your other linux compilers like cygwin or msys, change the name temporarily to force the make install to use DOS path naming):
If you configure PyQT and it spits out a file error that has to do with QTCore
Google pexports and download. Go to %QTdir%/bin. Then follow instructions or link ( http://jeethurao.com/blog/?p=18 )
pexports QtCore4.dll > QtCore4.def
dlltool –dllname QtCore4.dll –def QtCore4.def –output-lib libQtCore4.a
move libQtCore4.a ..\lib
And now you know kung-fu.
P.S: I never tried this method myself. This is a different, but untested (by me) method of compiling PyQT, done up by the trolls at Trolltech:
http://www.diotavelli.net/PyQtWiki/InstallingPyQTCommercialWin