CUDA - the launch timed out and was terminated - Ubuntu and no display - cuda

Question

I am using a workstation containing 4 GeForce GTX Titan black cards for CUDA development. I am working on Ubuntu 12.04.5 and none of these GPUs are used for display. I notice using cudaGetDeviceProperties that kernel execution timeout is enabled. Does this apply when I am not on Windows and not using a display?
I put the following code to test this in one of my kernels which normally runs fine:
__global__ void update1(double *alpha_out, const double *sDotZ, const double *rho, double, *minusAlpha_out, clock_t *global_now)
{
clock_t start = clock();
clock_t now;
for (;;) {
now = clock();
clock_t cycles = now > start ? now - start : now + (0xffffffff - start);
if (cycles >= 50000000000) {
break;
}
}
*global_now = now;
}
The kernel launch looks like:
update1<<<1, 1>>>(d_alpha + idx, d_tmp, d_rho + idx, d_tmp, global_now);
CudaCheckError();
cudaDeviceSynchronize();
For a large enough number of cycles waiting, I see the error:
CudaCheckError() with sync failed at /home/.../xxx.cu:295:
the launch timed out and was terminated
It runs fine for a small number of cycles. If I run this same code on a Tesla K20m GPU with kernel execution timeout disabled, I do not see this error and the program runs as normal. If I see this error, does it definitely mean I am hitting the kernel time limit that appears to be enabled or could there be something else wrong with my code? All mentions of this problem seem to be by people using Windows or also using their card for display so how is it possible I am seeing this error?

Answer 1

Linux has a display watchdog as well. On Ubuntu, in my experience, it is active for display devices that are configured via xorg.conf (e.g. /etc/X11/xorg.conf, but the exact configuration method will vary by distro and version).
So yes, it is possible to see the kernel execution timeout error on Linux.
In general, you can work around it in several ways, but since you have multiple GPUs, the best approach is to remove the GPUs that you want to do compute tasks on, from your display configuration (e.g. xorg.conf or whatever), and then run your compute tasks on those. Once X is not configured to use a particular GPU, that GPU won't have any watchdog associated with it.
Additional specific details are given here.
If you were to reinstall things, another approach that generally works to keep your compute GPUs out of the display path, is to load the Linux OS with the GPUs not plugged into the system. After things are configured the way you want display-wise, then add the compute GPUs to the system and load the linux toolkit. You will want to manually load the display driver instead of letting the linux toolkit do it, and deselect the option to have the linux display driver installer modify the xorg.conf This will similarly get your GPUs configured for compute usage but keep them out of the display path.