CUDA + CMake target library dependence breaks on different machine - cuda

Question

I recently tried to build my https://github.com/eyalroz/cuda-api-wrappers/ library's examples after switching to another Linux distribution on the same machine. Strangely enough, I encountered a linking issue. The command:
/usr/bin/c++ -Wall -std=c++11 -g CMakeFiles/device_management.dir/examples/by_runtime_api_module/device_management.cpp.o -o examples/bin/device_management -rdynamic lib/libcuda-api-wrappers.a -Wl,-Bstatic -lcudart_static -Wl,-Bdynamic -lpthread -ldl -lrt
fails to find the CUDA runtime library, and I get:
CMakeFiles/device_management.dir/examples/by_runtime_api_module/device_management.cpp.o: In function `cuda::device::peer_to_peer::get_attribute(cudaDeviceP2PAttr, int, int)':
/home/eyalroz/src/mine/cuda-api-wrappers/src/cuda/api/device.hpp:38: undefined reference to `cudaDeviceGetP2PAttribute'
collect2: error: ld returned 1 exit status
but if I add -L/usr/local/cuda/lib64 it builds fine. This didn't use to happen before; and it doesn't happen on another machine I've checked on, nor does it even happen to other targets using the CUDA runtime in the same CMakeLists.txt (like version_managament).
FindCUDA seems to be finding everything, as the value of ${CUDA_LIBRARIES} is /usr/local/cuda/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so. And the target lines in CMakeLists.txt are:
add_executable(device_management EXCLUDE_FROM_ALL examples/by_runtime_api_module/device_management.cpp)
target_link_libraries(device_management cuda-api-wrappers ${CUDA_LIBRARIES})
as is suggested in answers to other related questions (e.g. here). Why is this happening? Should I "manually" add the -L switch?
Edit: Following #RobertCrovella's suggestion, here are the ld search paths:
$ gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,; ,g' | tr \; \\012 | tr ':' "\n" | tail -n +3
/usr/local/cuda/lib64/x86_64-linux-gnu/5/
/usr/local/cuda/lib64/x86_64-linux-gnu/
/usr/local/cuda/lib/
/usr/lib/gcc/x86_64-linux-gnu/5/
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/5/
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/
/usr/x86_64-linux-gnu/lib/
/usr/lib/x86_64-linux-gnu/5/
/usr/lib/x86_64-linux-gnu/
/usr/lib/
/lib/x86_64-linux-gnu/5/
/lib/x86_64-linux-gnu/
/lib/
/usr/lib/x86_64-linux-gnu/5/
/usr/lib/x86_64-linux-gnu/
/usr/lib/
/usr/local/cuda/lib64/
/usr/x86_64-linux-gnu/lib/
/usr/lib/
/lib/
/usr/lib/
$ ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu")
SEARCH_DIR("=/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib")
Notes:
Yes, I know the CMakeLists.txt there is ugly.

Answer 1

TL;DR:
After the FindCUDA invocation, add the lines:
get_filename_component(CUDA_LIBRARY_DIR ${CUDA_CUDART_LIBRARY} DIRECTORY)
set(CMAKE_EXE_LINKER_FLAGS ${CMAKE_EXE_LINKER_FLAGS} "-L${CUDA_LIBRARY_DIR}")
and building should succeed on both systems.
Discussion:
(Paraphrasing #RobertCrovella and myself in the comments:)
OP was expecting, that if the following hold:
FindCUDA succeeds
${CUDA_LIBRARIES} includes a valid full path to either the static or the dynamic CUDA runtime library
the library dependency is indicated using target_link_libraries(relevant_target ${CUDA_LIBRARIES})
... then the CMake-based build he was attempting should succeed on a variety of valid CUDA installations. That is (unfortunately) not the case, since while FindCUDA does locate the CUDA library path, it does not actually make your linker search that path. So a failure should actually be expected. The build had worked on OP's old system due to a "fluke", or rather, due to OP having added the CUDA library directory to the linker's search path, somehow, apriori.
The linking command must be issued with the -L/path/to/cuda/libraries switch, so that the linker knows where to looks for the (unspecified-path) libraries referred to be the CUDA-related -l switches (in OP's case, -lcudart_static).
This answer discusses how to do that in CMake for different kinds of targets. You might also want to have a look at man gcc (the GCC manual page, also available here) regarding the -l and -L options, if you are not familiar with them.