MPI + CUDA software architecture on GPU cluster - cuda

Question

I would like to ask for an advice, when using OpenMPI and CUDA on GPU cluster.
I am a beginner and I feel I can't foresee consequences of my decisions about a software architecture. I would highly appreciate someone's advice/rule of thumb, as the information on GPU Clusters is quite sparse.
Framework:
Cluster architecture
the cluster has 1 front node and 9 computation nodes
the cluster is heterogeneous, every node has the Intel Xeon CPU(s) and Nvidia Tesla K80, but with different number of processors and different number of GPU cards
the cluster runs PBSPro scheduler
Goal
1) redistribute the data from a root_MPI_process to MPI_processes
2) load the data to GPU, execute kernel (SIMT-parallel calculations), get the results back
3) send the results back to root_MPI_process
4) root_MPI_process processes the results, creates new data
... iterate -> redistribute the data ...
The steps 1, 2, 3 are purely [SERIAL], and each spawned MPI_process independent from all the others, i.e. no pieces of data are moved between any two MPI_processes
My considerations for software architecture
Alt. 1) 1 MPI process == 1 GPU
start X MPI_processes, every MPI_process (except the root_MPI_process) is responsible for 1 GPU
the MPI_process then receives a chunk of data, suitable right-away to be passed to GPU and executes kernel ... steps described above
Alt. 2) 1 MPI process == 1 computational cluster node (with multiple GPUs)
start X MPI processes, every MPI_process (except the root_MPI_process) runs on 1 computational cluster node
the MPI_process then identifies number of GPUs, and asks for appropriate amount of data from root_MPI_process
the data, passed from root_MPI_process to the MPI_process, are then redistributed among available GPUs ... step 2, 3, 4 mentioned above
Questions
1) From experienced point of view, what else -- except the data passing (which is easier in 1) and more complicated in 2), from my point-of-view) -- should I consider ?
2) This application cannot take the advantage of CUDA aware MPI, because the data are not passed between GPUs, is that right ? ( Is CUDA aware MPI useful for something else then inter-GPU communication ? )
3) Solution 2) offers Universal Addressing Space with Single Address Space, but the solution 1) does not, because every MPI_process access 1 GPU, is that right ?
Edit
this is research in progress, and I don't dare to estimate E2E timing. For reference, this task takes approx. 60 hours on 3x GTX 1070, the cluster has 16x Tesla K80. My computational time at the moment is quite unlimited.
The data are approx 1 [kB] per thread, therefore 1 kernel requires blocks * threads * 1024 [B] of data, I would like to run 1 kernel per GPU at a time.
the kernel (each thread in each block) runs simulation of 2nd order dynamic system with evaluation of small neural network (30 neurons) (the number of multiplications and additions are in 100's per iteration), there are around 1,000,000 simulation iterations before delivering the result.
From the above I can say with confidence, that evaluation of the kernel is more time consuming than the data transfer from host<->device.

Answer 1

1) From experienced point of view, what else -- except the data passing (which is easier in 1) and more complicated in 2), from my point-of-view) -- should I consider ?
If your assumption of kernel execution time >>> communication time holds true, then this is a simple problem. Also if you don't benefit / intend to really utilize the Xeon CPUs, then things are simpler. Just use Alt. 1) (1 to 1, pure MPI). Alt. 2) means you would have to implement two tiers of workload distribution. There's no need to do that without a good reason.
If your assumptions don't hold true, things can get way more complicated and far beyond a concise answer on SO. Addressing these issues is not useful without a clear understanding of the application characteristics.
One thing that you may have to consider if your application runs for > 1 day, is checkpointing.
2) This application cannot take the advantage of CUDA aware MPI, because the data are not passed between GPUs, is that right ? ( Is CUDA aware MPI useful for something else then inter-GPU communication ? )
Since the CPU processes the resulting data in step 4), you wouldn't benefit from CUDA-aware MPI.
3) Solution 2) offers Universal Addressing Space with Single Address Space, but the solution 1) does not, because every MPI_process access 1 GPU, is that right ?
No, there are multiple (9) address spaces in your second approach. One address space per compute node. So you have to use MPI anyway, even in the second approach - which is exactly what makes the 1-rank-1-GPU mapping much simpler.
One thing you should consider, your step 4) will be come a scalability bottleneck at some point. But probably not at the scales you are talking about. It is worth investing in performance analysis tools / methodology to get a good understanding on how your code performs and where the bottlenecks are during development and scaling up to production.

Answer 2

I would start with the first alternative:
The data transfer to each node would be the same in either situation, so that's a, wash.
The first scenario lets the scheduler assign a core to each GPU with room to spare.
The time to spawn the multiple MPI listeners only occurs once if done right.
The second alternative has to, unless you add parallelism in each MPI worker, process each GPU in a serial fashion.
My only caveat is to watch the network and DMA for multiple cores fighting for data. If collisions dominate, add the extra code to implement the second alternative. There is little lost in coding the easier solution first and checking the first iteration at step 4 to see if data passing is problematic.