I developed an rtsp server to communicate with the vlc client.
The role of the server is as follows:
Connect the vlc client to the rtsp session.
It is to get the h264 payload from a specific pcap file and send it to the client with an rtp header.
It is well implemented, but there is one problem.
This means that the rtsp session must be terminated (TEARDOWN), but the decoding is initiated.
I hope to see the video in real time.
I want you to suggest a wise solution.
Attempted to raise the cache of the vlc client (30000ms)
The problem is still not solved, but I feel that the quality of the decoded image has improved.
I hope for a solution for real-time image decoding.
I solved the problem.
Turns out to be a timestamp issue in rtp packets.
When I sent the packet, I sent it as a realtime, but the interval was too short and the decoding was delayed.
When sending a packet, the problem is solved by forcing a suitable interval.
Related
I've been working on making a near real-time voice chat app. The webpage will send packets to the server, and the server will save the packets to disk, and then re transmit the packets to the other connected webpages. I've tried many other solutions, but they are either laggy or they do not play. I've realized that sending PCM samples would be optimal (the server will be recording these as well), but I'm not sure how to get them to play on another client's end. I'm using NodeJS with Socket.IO. Thanks in advance!
The webpage will send packets to the server, and the server will save the packets to disk, and then re transmit the packets to the other connected webpages.
Already, this is not that efficient. It's better when possible to send data directly from peer to peer.
I've realized that sending PCM samples would be optimal
No, it wouldn't. This requires more bandwidth, which is going to need better buffering, which means higher latency. This is voice chat... no need to use a lossless encoding like PCM.
I've been working on making a near real-time voice chat app.
This is basically the defacto primary use case that WebRTC was built for. If you use WebRTC, you get:
Peer-to-peer streaming (where possible)
NAT traversal (to enable those P2P connections, where possible, or proxy them when not)
Low latency optimization, from end to end
Hardware acceleration (where available)
Opus audio codec
Automatic resampling, for compatibility and to keep latency low as things drop
In other words, this is already a solved problem with WebRTC.
Is possible to pipe raw video frames to a browser/website? For instance the decoding could be done locally in Gstreamer, and then that could be forwarded somehow to a browser.
EDIT:
I realize that my description was a bit shaky. The use case I would like to have is to send encoded video to someone, decode it on their computer, do some advance filtering that cannot be done in the browser, and then pipe the frames to the browser. Obviously re-encoding it would just be a waste of time and energy.
All that I can find is ppl saying that video frames can be grabbed FROM a browser, no-one seems to be interested to SENT TO a browser. The horrible option could be to use webrtc and to re-encode the frames into VP8 and then to send it to the browser.
So my final question is whether it is possible to write to the rendering pipeline of a browser? I know next to nothing about web programming, I usually just deal with images and video.
Thank you for your support :)
PS: forgive my lack of knowledge, is it possible to have a client on someone's computer, writting to a local tcp port, and to access that tcp port from a website in the browser? (potentially asking the user to allow the connection?)
Yes, this is possible. Since you're running a local GStreamer pipeline, you might look into this project: https://github.com/Samsung/ChromiumGStreamerBackend Basically, they're using GStreamer as the native renderer in-browser.
Aside from that, you can create a browser extension which executes an application and gets data from Gstreamer, to shuffle to your application. https://developer.chrome.com/extensions/nativeMessaging
If you don't want to make an extension, you can instead create a small Web Socket server.
Either way, you can write the raw pixel data to a Canvas... no need to re-encode/decode the video. https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API
I'd like to stream a user's webcam (from the browser) to a server and I need the server to be able to manipulate the stream (run some C algorithms on that video stream) and send the user back information.
I have heavily looked at WebRTC and MediaCapture and read the examples here : https://bitbucket.org/webrtc/codelab/overview .
However this is made for peer-to-peer video chat. From what I have understood, the MediaStream from getUserMedia is transmitted via a RTCPeerConnection (with addStream) ; what I'd like to know is : can I use this, but process the video stream on the server ?
Thanks in advance for your help
Here is the solution I have designed.
I post here for people seeking the same kind of information :-)
Front End side
I use the WebRTC API : get webcam stream with getUserMedia, open RTCPeerConnection (and RTCDataChannel for downside information).
The stream is DTLS encrypted (mandatory), multimedia streams use RTP and RTCP. The video is VP8 encoded and the audio in Opus encoded.
Back End side
On the backend, this is the complex part.
The best (yet) alternative I could find is the Janus Gateway. It takes cares of a lot of stuff, like DTLS handshake, RTP / RTCP demuxing, etc. Basically, it fires a event each time a RTP packet is transmitted. (RTP packets are typically the size of the MTU, so there is not a 1:1 mapping between video frames and RTP packets).
I then built a GStreamer (version 1.0) to depacketize the RTP packets, decode the VP8, ensure video scaling and colorspace / format conversion to issue a BGR matrix (compatible with OpenCV). There is an AppSrc component at the beginning of the pipeline and a AppSink at the end.
What's left to do
I have to take extra measures to ensure good scalability (threads, memory leaks, etc) and find a clean and efficient way of using the C++ library I have inside this program.
Hope this helps !
How do I achieve streaming of audio and video data and pass it on the network. I gone through a good article Here, But did not get in depth. I want to have chat application in HTML5
There are mainly below question
How to stream the audio and video data
How to pass to particular IP address.
Get that data and pass to video and audio control
If you want to serve a stream, you need a server doing so, by either downloading and installing, or coding on your own.
Streams only work in one direction, there is no responding or "retrieve back". Streaming is almost the same as downloading, with slight differences, depending on the service and use case.
Most streams are downstreams, but there are also upstreams. Did you hear about BufferStreams in PHP, Java, whatever? It's basically the same: data -> direction -> cursor.
Streams work over many protocols, even via different network layers, for example:
network/subnet broadcast, peer 2 peer, HTTP, DLNA, even FTP streams, ...
The basic nature of a stream is nothing more than data beeing sent to an audience.
You need to decide:
which protocol do you want to use for streaming
which server software
which media / source / live or with selectable start/end
which clients
The most popular HTTP streaming server is Shoutcast by Nullsoft (Winamp).
There is also DLNA which afaik is not HTTP based.
To provide more information, you need to be more specific regarding your basic requirements and decisions.
I am trying to learn enough about H.264, RTP, RTSP and encapsulation file formats to develop a video recording application.
Specifically, what should I read to understand the problem?
I want to be able to answer the following questions:
Can I save H.264 packets or NALs (Per RFC 6184) to a file?
Can I save the individual payloads as files?
Can I join the RTP payloads simply by concatenating them?
What transformation is needed to save
several seconds of H.264 video in an MP4 container.
What must be done
to later join these MP4 files, or arbitrarily split them, or serve
them as a new RTSP presentation?
I want to be able to answer these questions on a fairly low level so I can implement software that does some of the processes (capture RTP streams, rebroadcast joined MP4s).
Background
The goal is to record video from a network camera onto disk. The camera has an RTSP server that provides an H.264 encoded stream which it sends via RTP to a player. I have successfully played the stream using VLC, but would like to customize the process.
The "raw" video stream is a sequence of NAL units, per H.264 specification. Neither on RTSP, nor on MP4 file you have this stream "as is".
On RTSP connection you typically receive NAL units fragmented, and you need to depacketize them (no you cannot simply concatenate):
RTP H.264 Packet Depacketizer
How to process raw UDP packets so that they can be decoded by a decoder filter in a directshow source filter
MP4 file is a container formatted file, and has its own structure (boxes). So you cannot simply stream NALs into such file and you have to do what is called multiplexing.
How do I create an mp4 file from a collection of H.264 frames and audio frames?
just install rtmpdump along with rtmpsrv and rtmpsuck...
this will do all the work
in one terminal open rtmpsrv and in other open rtmpdump -r "RTMP URL"
this will save the stream in mystream.flv