CUDNN_STATUS_MAPPING_ERROR when training with pose2body - deep-learning

Question

I'm trying to train https://github.com/NVIDIA/vid2vid. I'm...
...executing with pretty much the vanilla parametrization shown in the readme, I had to change the number of GPUs though and increased the number of threads for reading the dataset. Command:
python train.py \
--name pose2body_256p \
--dataroot datasets/pose \
--dataset_mode pose \
--input_nc 6 \
--num_D 2 \
--resize_or_crop ScaleHeight_and_scaledCrop \
--loadSize 384 \
--fineSize 256 \
--gpu_ids 0,1 \
--batchSize 1 \
--max_frames_per_gpu 3 \
--no_first_img \
--n_frames_total 12 \
--max_t_step 4 \
--nThreads 6
...training on the supplied example datasets.
...running a docker container built with the scripts in vid2vid/docker, e. g. with CUDA 9.0 and CUDNN 7.
...using two NVIDIA V100 GPUs.
Whenever I start training the script crashes after a couple of minutes with the message RuntimeError: CUDNN_STATUS_MAPPING_ERROR. Full error message:
Traceback (most recent call last):
File "train.py", line 329, in <module>
train()
File "train.py", line 104, in train
fake_B, fake_B_raw, flow, weight, real_A, real_Bp, fake_B_last = modelG(input_A, input_B, inst_A, fake_B_last)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
raise output
File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
output = module(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/vid2vid/models/vid2vid_model_G.py", line 130, in forward
fake_B, fake_B_raw, flow, weight = self.generate_frame_train(netG, real_A_all, fake_B_prev, start_gpu, is_first_frame)
File "/vid2vid/models/vid2vid_model_G.py", line 175, in generate_frame_train
fake_B_feat, flow_feat, fake_B_fg_feat, use_raw_only)
File "/vid2vid/models/networks.py", line 171, in forward
downsample = self.model_down_seg(input) + self.model_down_img(img_prev)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR
From reading the issues in the vid2vid using two V100 should work with this setup. The error also occurs if CUDA 8/CUDNN 6 are used. I checked the flags but haven't found any indication of further necessary changes to the arguments supplied to train.py.
Any ideas on how to solve (or work around) this?

Answer 1

In case anybody deals the same issue: Training on P100 cards worked. Seems like the V100 architecture clashes with version of pytorch used in the supplied Dockerfile at some point. Not quite a solution, but a workaround.