Use Media Foundation H.264 encoder with live555 - h.264

Question

I want to create an H.264 RTSP stream using the live555 streaming library. For encoding the video frames, I want to use the H.264 encoder MFT. Encoding works using the basic processing model (I do not build a graph, but call the MFT manually). Streaming using a custom FramedSource source also seems to work in the sense that the programme is not crashing and the stream is stable in VLC player. However, the image is crippled - no colour, weird line patterns etc.
I assume that I pass the wrong data from the encoder into the streaming library, but I have not been able to find out what the library is actually expecting. I have read that the Microsoft H.264 encoder outputs more than one NAL in a sample. I further found that live555 requires a single NAL to be returned in doGetNextFrame. Therefore, I try to identify the individual NALs (What does this H264 NAL Header Mean? states that the header can be 3 or 4 bytes - I do not know where to get the information what MF uses, but the memory view of the debugger suggests 4 bytes):
for (DWORD i = 0; i < sampleLen; ++i) {
auto v = *reinterpret_cast<unsigned int *>(sampleData + i);
if (v == ::htonl(1)) {
nals.push_back(sampleData + i);
}
}
This piece of code usually identifies more than one item in one output sample from the MFT. However, if I copy the ranges found by this loop into the fTo output buffer, VLC does not show anything and stops after a few seconds. I also read somewhere that live555 does not want the magic number 0x00000001, so i tried to skip it. The effect on the client side is the same.
Is there any documentation on what live555 expects me to copy into the output buffer?
Does the H.264 encoder in Media Foundation at all produce output samples which I can use for streaming?
Do I need to split the output samples? How much do I need to skip once I have found a magic number (How to write a Live555 FramedSource to allow me to stream H.264 live suggests that I might need to skip more than the magic number, because the accepted answer only passes the payload part of the NAL)?
Is there any way to test whether the samples returned by the H.264 MFT in basic processing mode form a valid H.264 stream?

Answer 1

Here's how I did it MFWebCamRtp.
I was able to stream my webcam feed and view it in VLC. There was no need to dig into NALs or such. Each IMFSample from the Media Foundation H264 encoder contains a single NAL that can be passed straight to live555.