CUDA - Which stream will child grid be in? - cuda

Question

If I use dynamic parallelism, which stream will child grid be running in?
For example, I have one kernel called A, and another kernel called B. B is launched by A.
If kernel A is running in stream_A, and also if I does not specify the stream ID for kernel B, which stream will the kernel be running in? Is it the default stream, or it will inherit the stream A is running in?

Answer 1

Based on what I see in the documentation, as well as my own experiments, what I observe is that in general there is no (ordering) relationship between streams on the host and streams on the device.
We can consider two cases:
First case, for created streams, I would say this is covered explicitly in the documentation.
In the second case, although the documentation mentions the device NULL stream, that is perhaps a bit unclear. We can write a simple test case to sort this out for us:
$ cat cdp.cu
#include <cstdio>
#include <cassert>
#include <unistd.h>
const unsigned long long dt = 10000000000ULL;
__device__ void delay(){
unsigned long long start = clock64();
while (start+dt > clock64());
}
__global__ void child(){
delay();
}
__global__ void parent(){
child<<<1,1>>>();
cudaError_t err = cudaGetLastError();
assert(err == cudaSuccess);
err = cudaDeviceSynchronize();
assert(err == cudaSuccess);
}
__global__ void pk(){
printf("hello\n");
}
int main(){
cudaStream_t s1, s2;
cudaStreamCreate(&s1);
cudaStreamCreate(&s2);
parent<<<1,1, 0, s1>>>();
sleep(1);
pk<<<1,1,0,s2>>>();
cudaDeviceSynchronize();
}
$ nvcc -o cdp cdp.cu -rdc=true -lcudadevrt -lineinfo
$ time ./cdp
hello
real 0m7.276s
user 0m4.193s
sys 0m2.061s
$
(Ubuntu 18.04, CUDA 11.4, V100)
When I run the above code, I observe that after launching the cdp executable, the console remains idle for approximately 1 second. After that the console prints "hello", and then the console remains idle for 6-7 seconds, and then the application exits. (Please don't expect the same behavior in a WDDM GPU.)
If the NULL stream on the device were the "same as" or "inherited from" the NULL stream on the host, then we would expect that the "hello" printout would not appear until immediately before application exit. The pk kernel would not be allowed to run until the child kernel had completed, if host NULL stream semantics were involved. Therefore we must conclude that the device NULL stream is not the same as the host NULL stream.
We can use a little bit of logic to convince ourselves that this is also not exactly the same as inheriting the host (created) stream used for the parent kernel launch. We can read in the documentation that the use of the NULL stream for child kernel launches within the same threadblock will have the NULL stream behavior. However this could not be true if the device NULL stream were simply the same as/inherited from the host created stream. According to the documentation, if I have two child kernel launches from the same threadblock, one into the NULL stream, and another into a device created stream, then we would not expect these to be able to run concurrently - that is the behavior of the NULL stream. But if the NULL stream were simply the inherited host stream, a created stream, there is no reason to expect that they could not run concurrently.
So we are left with the conclusion that the device NULL stream is not the host NULL stream, and the device NULL stream is also not a host created stream. These statements seem consistent with the documentation to me.
If you would like a clarification in the documentation, the usual suggestion is to file a bug.
Rather than worry about peculiarities of NULL stream behavior, the advice I give when teaching CUDA is that if you are concerned about complex concurrency scenarios, do all your work with created streams. Leave the NULL stream behind. Anything you wish to do can be done purely with created streams. Do as you wish, of course.