Capture system sound from browser - google-chrome

I am trying to build a web app that captures both local and remote audios from a webrtc call, but I can`t record the remote audio(using recordRTC).
I was wondering if I could capture the system sound somehow.
Is there a way to capture the system sound (not just the mic) from the browser. Maybe an extension?

In Chrome, the chrome.desktopCapture extension API can be used to capture the screen, which includes system audio (but only on Windows and Chrome OS and without plans for OS X or Linux). E.g.
chrome.desktopCapture.chooseDesktopMedia([
'screen', 'window' // ('tab' is not supported; use chrome.tabCapture instead)
], function(streamId) {
navigator.webkitGetUserMedia({
audio: {
mandatory: {
chromeMediaSource: 'system',
chromeMediaSourceId: streamId
}
},
video: false, // We only want audio for now.
}, function(stream) {
// Do what you want with this MediaStream.
}, function(error) {
// Handle error
});
});
I'm not sure whether Firefox can capture system sound, but at the very least it is capable of capturing some output (tab/window/browser/OS?).
First you need to visit about:config and set media.getusermedia.audiocapture.enabled to true (this could be automated through a Firefox add-on). Then the stream can be captured as follows:
navigator.mozGetUserMedia({
audio: {
mediaSource: 'audioCapture'
},
video: false, // Just being explicit, we only want audio for now
}, function(stream) {
// Do what you want with this MediaStream.
}, function(error) {
// Handle error
});
This was implemented in Firefox 42, at https://bugzilla.mozilla.org/show_bug.cgi?id=1156472

This is possible with the new Screen Capture API, but browser support is still limited.
See the "Browser compatibility" section in the above-linked MDN page for details. Some browsers currently don't yet support audio capture, and some others currently only allow audio capture from a specific tab, rather than the operating system as a whole.
Example code:
videoElem.srcObject = await navigator.mediaDevices.getDisplayMedia({audio:true, video:true});

Related

MediaRecorder does not produce a valid WebM file

MediaRecorder.isTypeSupported() returns true for 'video/webm', 'video/webm;codecs=avc1', and 'video/webm;codecs=vp9'.
However, only mimeType='video/webm;codecs=vp9' results in a valid webm file ('video/webm;codecs=vp9,opus').
Setting mimeType to either 'video/webm' and 'video/webm;codecs=avc1' results in 'video/x-matroska;codecs=avc1,opus' file, which is not a valid webm video.
Is this a MediaRecorder bug, Chrome bug or am I missing something?\
Software: Chrome 85, MacOS 10.15. I have not tried this on Windows or Linux.
Reproduction:
const wantMimeType = 'video/webm;codecs=h264,opus';
if (MediaRecorder.isTypeSupported(wantMimeType)) {
let mediaRecorder = new MediaRecorder(stream, {
mimeType: wantMimeType,
});
// ...drive the recorder
mediaRecorder.onstop = (event) => {
try {
// returns 'video/x-matroska;codecs=avc1,opus' in Chrome,
// and the binary file content also says Matroska instead of WebM
let mimetype = mediaRecorder.mimeType;
if (!mimeType.startsWith('video/webm')) {
throw new Error(`We requested "${wantMimeType},
but the browser gave us "${mediaRecorder.mimeType}"`);
}
let blob = new Blob(chunks, { type: mimeType });
// ...convert to data: URL
// ...play it in <video src="data:..."> // won't work, if Matroska
} catch (ex) {
alert(ex);
}
};
}
Observations:
Chrome responds true to isTypeSupported('video/webm;codecs=h264,opus'), but Chrome actually does not support this combination, i.e. the response is factually incorrect and defeats the entire purpose of isTypeSupported().
Chrome ignores the mimetype that we pass in the MediaRecorder constructor, and instead returns video/x-matroska;codecs=avc1,opus. Which then doesn't work, because <video> won't play the Matroska mimetype.
2 solutions:
Hacky: After recording, just change the mimetype back to video/webm;codecs=h264,opus. The file content will still say "Matroska", but <video> on Chrome will play it. It's a bad solution, because we're passing in the wrong mimetype, it doesn't match the content. Other software - or Chrome in later versions - might break over it.
Proper: Don't use H.264, but VP9 or VP8. Chrome supports both, and Firefox supports VP8, and they work properly. As a bonus, they are open, while H.264 is riddled with software patents. Depending on the hardware encoder implementation, VP8/9 also may give better image quality results.
However, only mimeType='video/webm;codecs=vp9' results in a valid webm file.
Is this a MediaRecorder bug, Chrome bug or am I missing something?
WebM is the correct a/v media container for the VP8 and VP9 video codecs.
AVC1 video codec belongs into an MPEG container (.mp4, .m4v).
This video codec can also be contained inside an MKV media file (Matroska .mkv).

Cannot replay MP3 in Firefox using MediaSource even though it works in Chrome

I have implemented a simple audio player in my web application and noticed that it is not working in Firefox (let me just ... 🎉).
What I get is an error:
ERROR DOMException: MediaSource.addSourceBuffer: Type not supported in MediaSource
This is followed by a warning:
Cannot play media. No decoders for requested formats: audio/mpeg
This is the implementation for the sourceopen event handler:
private onSourceOpen = (e) => {
this.logger.debug('onSourceOpen');
if (!this.sourceBuffer) {
this.sourceBuffer = this.mediaSource.addSourceBuffer('audio/mpeg');
}
this.mediaSource.removeEventListener('sourceopen', this.onSourceOpen);
this.fetchRange(this.trackPlayUrl, 0, this.segmentLength, (chunk) => this.appendSegment(chunk));
}
Where
// Create the media source object
this.mediaSource = new MediaSource();
this.mediaSource.addEventListener('sourceopen', this.onSourceOpen);
Why does it hate me?
Before you try to create a SourceBuffer, you should always call MediaSource.isTypeSupported to determine whether it is likely to play. If that returns false, the user agent is telling you it definitely won't work.
On the latest Firefox:
>> MediaSource.isTypeSupported('audio/mpeg')
<- false
It hates you because Firefox's MediaSource implementation can't play content with that MIME type, whereas Chrome's can.
AAC in ISOBMFF has very broad support, though this would require transcoding and repackaging your audio - try:
MediaSource.isTypeSupported('audio/mp4; codecs="mp4a.40.2"')

webrtc configuration to reduce sent traffic

I am developing the function of audio video calls using webrtc technology. I faced with case when one of the participants is in an unstable network (mobile network). In this case, the participant’s audio and video starts to freeze, there is a delay, etc. I think that to solve this problem it is necessary to configure the application so that it sends less traffic.
Please tell me what webrtc configurations exist to reduce sent traffic?
Most heavy traffic will be video. One of the solutions will be limiting video quality or disabling it completely. You can limit video quality using this code:
const displayMediaStream = await getDisplayMediaStream();
let supports = navigator.mediaDevices.getSupportedConstraints();
if (!supports["width"] || !supports["height"] || !supports["frameRate"] || !supports["facingMode"]) {
// We're missing needed properties, so handle that error.
} else {
let constraints = {
width: { min: 640, ideal: 1920, max: 1920 },
height: { min: 400, ideal: 1080 },
aspectRatio: 1.777777778,
frameRate: { max: 30 }
};
displayMediaStream.getVideoTracks()[0].applyConstraints(constraints)
}
return displayMediaStream.getVideoTracks()[0];
You can play with the values.
Also, the problem could be in the browser codec. For example, FF in case of screen sharing uses codec which produces high-quality video stream which is good in case of static pictures, like sharing documents, the problem appears when users broadcast dynamic videos, like youtube videos, with screen sharing. In such a case, FF overloads the network by sending streams ~7 Gb. Meanwhile, Google Chrome is more intelligent and can adapt the traffic by using better codecs. I would do tests with different browsers and if the problem lays in FF, you can try to force FF to use better codecs, like same which used by Google Chrome, for that you have to modify SDP when detecting FF browser, you can do it like described here: How can I change the default Codec used in WebRTC?

Html5 amp story convert to mo4 video

What is best practice to convert html5 and css3 animation or amp-stories to mp4 video?
Is there any library which can do this out of the box?
What I've done in the past is utilize Chrome's Tab Capture API.
https://developers.chrome.com/extensions/tabCapture
Basically, you build a browser extension and create a media stream and Media Recorder. Something like this:
const mediaStream = chrome.tabCapture.captureOffscreenTab('https://example.com', {
audio: true,
video: true,
videoConstraints: {
width: 1280,
height: 960
}
});
Then, use MediaRecorder to capture the output and save it, in the same way you would any other MediaStream.

WebRTC - disable all audio processing

I'm currently trying to get a clean as possible audio channel via webrtc. Via the getUserMedia mediaconstraints object, I've set the following options:
constraints: {
audio: {
mandatory: {
echoCancellation: false,
googEchoCancellation: false,
googAutoGainControl: false,
googAutoGainControl2: false,
googNoiseSuppression: false,
googHighpassFilter: false,
googTypingNoiseDetection: false,
//googAudioMirroring: false // For some reason setting googAudioMirroring causes a navigator.getUserMedia error: NavigatorUserMediaError
}
},
video: false
},
This greatly improves the audio quality, but there still seems to be audio processing present which causes the mutilation of the audio in the form of high frequency noise with some of the test samples.
There is a Chrome flag --use-file-for-fake-audio-capture as described at http://peter.sh/experiments/chromium-command-line-switches/#use-file-for-fake-audio-capture which allows input via file for testing. As mentioned in the description of the flag, all audio processing has to be disabled or the audio will be distorted - so there seems to be additional options for this purpose.
I also tried the --disable-audio-track-processing --audio-buffer-size=16 --enable-exclusive-audio Chrome flags, but still there seems to be some audio processing.
Is there any way to disable the still present audio processing (preferably via JS API)?
I'd wager that the variable bitrate (default) behavior of the opus codec is causing some compression or adjustment. You could manually mangle the SDP offer to use CBR (constant bitrate) instead of VBR (variable bit rate). When you get the SDP offer from the browser, change the line:
a=fmtp:111 minptime=10; useinbandfec=1
to:
a=fmtp:111 minptime=10; cbr=1
Note that I'm both adding cbr=1 and removing useinbandfec=1. I'm not positive that dropping useinbandfec is necessary, but it seems that in-band FEC (forwarding error correction) causes compression adjustment which you'd want to avoid as well.
This is the updated way to disable audio processing and get a clean signal:
navigator.mediaDevices.getUserMedia({
audio: {
autoGainControl: false,
channelCount: 2,
echoCancellation: false,
latency: 0,
noiseSuppression: false,
sampleRate: 48000,
sampleSize: 16,
volume: 1.0
}
});
If you are streaming audio via WebRTC, it defaults to radio or phone quality audio optimized for voice. So make sure your SDP has stereo and maxaveragebitrate params:
a=fmtp:111 minptime=10;useinbandfec=1; stereo=1; maxaveragebitrate=510000