I have a simple setup for Desktop Capturing using html5 libraries.
This includes a simple webpage and a chrome-extension. I am using
Extension to get the sourceId
Using the sourceId I call navigator.mediaDevices.getUserMedia to get the MediaStream
This MediaStream is then fed into an instance of MediaRecorder for recording.
This setup works most of the times, but a few times I see that requestData() on MediaRecorder instance returns blob with empty data consistently. I am clueless as to what can cause a running setup to start misbehaving sometimes.
Some weird behaviour that I noticed in the bad state:
When I try to close/refresh the window it doesn't respond.
The MediaStreamTrack object in Step 2) above is 'live' but as soon as I go to Step 3) it becomes 'muted'.
There's no pattern to it, sometimes it even happens when I request for the MediaStreams the very first time(which rules out the possibility that there could be some dangling resources eating up the contexts)
Is there anything that I am doing wrong and am unaware of? Any help/pointers would be highly appreciated!
Related
I am trying a build a realtime monitoring system for high frequency data. To increase the performance, I used the extendData property of dcc.Graph() and websocket. So that, the brouser does not need to send request to get data.
I found that it still not increasing the performance as expected. The reason I found is, from the browser, I see (by inspecting network from browser) after some miliseconds browser is still sendng request and the initiator is the dash_renderer.
This picture is for a vanilla example just to show even for a textbox example the http request goes on and on. And for my real time websocket dashboard the frequency of requests get very high.
My question is:
What dash_renderer do?
why it is sending http request?
And how to stop that?
If you run Dash in Debug mode, it has a feature called Hot Reloading which regularly (every 3 seconds by default) checks for changes to your codebase and updates your running app if it finds any. That check for updated code is what you're seeing in the network inspection.
To turn it off, either don't run in debug mode or explicitly set dev_tools_hot_reload to False like so:
app.run_server(debug=True, dev_tools_hot_reload=False)
Although it is late, After some experience, my realization is dash is not designed to work with websocket. It uses call-backs which actually sends requests to server and in server, the callback function (which is python) send back some result.
These call-backs are designed to send HTTP request to server.
For high speed data, the websocket should be used with extendTrace method of plotly.js in client side.
I currently try to build a webrtc application that uses the RTCRtpTransceiver-Objects of the WebRTC standard. I add video and audio tracks to the connection and then, some time later, I try to remove them.
I do this with the following lines:
// part of a method, that searches the transceiver to stop and assigned it to the 'transceiver' variable
peerconnection.removeTrack(transceiver.sender);
if (transceiver.direction === "sendrecv") transceiver.direction = "recvonly";
else transceiver.direction = "inactive";
I know that this will set the remote track just to muted state and is like replacing the transceivers sender track with null, but Chrome has not implemented direction="stopped" or transceiver.stop() yet, see Issue 980879.
So what to do instead?
The remote peer just sees the current direction of the receiver becoming inactive and the receiver track gets muted.
Ergo, it does not remove the track for the remote peer in my application and muted tracks accumulate over time and even worse for video tracks: they get displayed like normal muted tracks.
I also cannot remove every muted track, since my application is required to allow muted and unmuted tracks (- muted by the user, not muted due to being stopped).
Muting tracks by entirely removing them will lead to another offer-answer exchange and this takes some time. I would prefer to just mute by replacing the track with null (without having to send sdp back and forth), as the WebRTC standard seems to allow and Chrome already implements.
The next option would be correlating the transceivers on both ends and sending a message from the stopping peer to the remote one to make the remote one stop the received track. This may be possible (I believe the transceivers mid should be the same, according to the spec), but in my opinion it is also an ugly way.
What i cannot do is just sending over the media track id, since this one is different for the stopping peer and the remote peer, so I cannot just send a message like 'please stop your track with id xyz' (it would make things easy, but well, it just doesn't work like this).
So, now I have the following questions:
What is the current 'standard' way in which you and other webrtc developers solve this, as long stopping transceivers will not work in every browser? (chrome fixing it for current versions will probably take some time but being the most used browser, we cannot just ignore it)
Did someone somehow (by abusing DTFM, using magic or whatever...) bodged a polyfill together? (I believe Adapter.js hasn't made this somehow possible)
Is there another way besides muting=removing and sending remote-track-stop messages over the signaler? If not, what is the better option?
What I ended up doing is choosing the signaling-solution.
I basically do
// part of a method, that searches the transceiver to stop and assigned it to the 'transceiver' variable
peerconnection.removeTrack(transceiver.sender);
if (transceiver.direction === "sendrecv") transceiver.direction = "recvonly";
else transceiver.direction = "inactive";
signaler.send({type: 'transceiver:stop', data: transceiver.mid});
And on the other side
// received sent message and called this method with message.data
onTransceiverStopMessage(mid){
const transceiverToStop = peerconnection.getTransceivers().filter(transceiver => transceiver.mid === mid);
transceiverToStop.receiver.track.stop()
transceiverToStop.receiver.track.dispatchEvent(new Event('ended'));
}
The dispatchEvent-Call is necessary, since calling stop on a track does not trigger the ended-Event.
It works well for my purpose, but I am still waiting for a solution to the piling up inactive transceivers (since they still impact performance, as the chrome issue states)
Is it possible for a Chrome extension to listen for streaming audio from any of the browser's tabs? I would like to capture the streaming audio data and then analyse it.
Thanks
You could try 3 ways, neither one does provide 100% guarantee to meet your needs.
Before going into more detailed descriptions, I must note that Chrome extensions do not provide convenient tools for working on per connection level - sufficiently low level, required for stream capturing. This is by design. This is why the 1-st way is:
To look at other browsers, for example Firefox, which provides low-level APIs for connections. They are already known to be used by similar extensions. You may have a look at MediaStealer. If you do not have a specific requirement to build your system on Chrome, you should possibly move to Firefox.
You can develop a Chrome extension, which intercepts HTTP-requests by means of webRequest API, analyses their headers and extracts media urls (such as containing audio/mpeg MIME-type, for example, in HTTP-headers). Just for a quick example of code you make look at the following SO question - How to change response header in Chrome. Having the url you may force appropriate media download as a file. It will land in default downloads folder and may have unfriendly name. (I made such an extension, but I do not have requirements for further processing). If you need to further process such files, it can be a challenge to monitor them in the folder, and run additional analysis in a separate program.
You may have a look at NPAPI plugins in general, and their streaming APIs in particular. I can imagine that you create a plugin registered for, again, audio/mpeg MIME-type, and receives the data via NPP_NewStream, NPP_WriteReady and NPP_Write methods. The plugin can be wrapped into a Chrome extension. Though I made NPAPI plugins, I never used this API, and I'm not sure it will work as expected. Nethertheless, I'm mentioning this possibility here for completenees. This method requires some coding other than web-coding, meaning C/C++. NB. NPAPI plugins are deprecated and not supported in Chrome since September 2015.
Taking into account that you have some external (to the extension) "fingerprinting service" in mind, which sounds like an intelligent data processing, you may be interested in building all the system out of a browser. For example, you could, possibly, involve a HTTP-proxy, saving media from passing traffic.
If you're writing a Chrome extension, you can use the Chrome tabCapture API to record audio.
chrome.tabCapture.capture({audio: true}, function(stream) {
var recorder = new MediaRecorder(stream);
[...]
});
The rest is left as an exercise to the reader; MDN has more documentation on how to use MediaRecorder.
When this question was asked in 2013, neither chrome.tabCapture nor MediaRecorder existed.
Mac OSX solution using soundflower: http://rogueamoeba.com/freebies/soundflower/
After installing soundflower it should appear as a separate audio device in the sound preferences (apple > system preferences > sound). Divert the computer's audio to the 2ch option (stereo, 16ch is surround), then inside a DAW, such as 'audacity', set the audio input as soundflower. Now the sound should be channeled to your DAW ready for recording.
Note: having diverted the audio from the internal speakers to soundflower you will only be able to hear the audio if the 'soundflowerbed' app is actually open. You know it's open if there's a 8 legged blob in the top right task bar. Clicking this icon gives you the sound flower options.
My privoxy has the following log:
2013-08-28 18:25:27.953 00002f44 Request: api.audioaddict.com/v1/di/listener_sessions.jsonp?_method=POST&callback=_AudioAddict_WP_ListenerSession_create&listener_session%5Bid%5D=null&listener_session%5Bis_premium%5D=false&listener_session%5Bmember_id%5D=null&listener_session%5Bdevice_id%5D=6&listener_session%5Bchannel_id%5D=178&listener_session%5Bstream_set_key%5D=webplayer&_=1377699927926
2013-08-28 18:25:27.969 0000268c Request: api.audioaddict.com/v1/ping.jsonp?callback=_AudioAddict_WP_Ping__ping&_=1377699927928
2013-08-28 18:25:27.985 00002d48 Request: api.audioaddict.com/v1/di/track_history/channel/178.jsonp?callback=_AudioAddict_TrackHistory_Channel&_=1377699927942
2013-08-28 18:25:54.080 00003360 Request: pub7.di.fm/di_progressivepsy_aac?type=.flv
So I got the stream url and record it:
D:\Profiles\user\temp>wget pub7.di.fm/di_progressivepsy_aac?type=.flv
--18:26:32-- http://pub7.di.fm/di_progressivepsy_aac?type=.flv
=> `di_progressivepsy_aac#type=.flv'
Resolving pub7.di.fm... done.
Connecting to pub7.di.fm[67.221.255.50]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [video/x-flv]
[ <=> ] 1,234,151 8.96K/s
I got the file that can be reproduced in any multimedia pleer.
I'm trying to capture audio using WASAPI. My code is largely based on the ChatterBox VoIP sample app. I'm getting audio buffers, but they are all silent (flagged AUDCLNT_BUFFERFLAGS_SILENT).
I'm using Visual Studio Express 2012 for Windows Phone. Running on the emulator.
I had the exact same problem and managed to reproduce it in the ChatterBox sample app if I set Visual Studio to native debugging and at any point stepped through the code.
Also, closing the App without going through the "Stop" procedure and stopping the AudioClient will require you to restart the emulator/device before being able to capture audio data again.
It nearly drove me nuts before I figured out the before mentioned problems but I finally got it working.
So..
1. Be sure to NOT do native debugging
2. Always call IAudioClient->Stop(); before terminating the App.
3. Make sure you pass the correct parameters to IAudioClient->Initialize();
I've included a piece of code that works 100% of the time for me. I've left out error checking for clarity..
LPCWSTR pwstrDefaultCaptureDeviceId =
GetDefaultAudioCaptureId(AudioDeviceRole::Communications);
HRESULT hr = ActivateAudioInterface(pwstrDefaultCaptureDeviceId,
__uuidof(IAudioClient2), (void**)&m_pAudioClient);
hr = m_pAudioClient->GetMixFormat(&m_pwfx);
m_frameSizeInBytes = (m_pwfx->wBitsPerSample / 8) * m_pwfx->nChannels;
hr = m_pAudioClient->Initialize(AUDCLNT_SHAREMODE_SHARED,
AUDCLNT_STREAMFLAGS_NOPERSIST | AUDCLNT_STREAMFLAGS_EVENTCALLBACK,
latency * 10000, 0, m_pwfx, NULL);
hr = m_pAudioClient->SetEventHandle(m_hCaptureEvent);
hr = m_pAudioClient->GetService(__uuidof(IAudioCaptureClient),
(void**)&m_pCaptureClient);
And that's it.. Before calling this code I've started a worker thread that will listen to m_hCaptureEvent and call IAudioCaptureClient->GetBuffer(); whenever the capture event is triggered.
Of course using Microsoft.XNA.Audio.Microphone works fine to, but it's not always an option to reference the XNA framework.. :)
It was a really annoying problem which waste about 2 complete days of mine.My problem was solved by setting AudioClientProperties.eCatagory to AudioCategory_Communications instead of AudioCategory_Other.
After this long try and error period I am not sure that the problem won't repeat in the future because the API doesn't act very stable and every run may return a different result.
Edit:Yeah my guess was true.Restarting the wp emulator makes the buffer silent again.But changing the AudioClientProperties.eCatagory back to AudioCategory_Other again solve it.I still don't know what is wrong with it and what is the final solution.
Again I encounter the same problem and this time commenting (removing) the
properties.eCategory = AudioCategory_Communications;
solve the problem.
I can add my piece of advice for Windows Phone 8.1.
I made the following experiment.
Open capture device. Buffers are not silent.
Open render device with AudioDeviceRole::Communications. Buffers immediately go silent.
Close render device. Buffers are not silent.
Then I opened capture device with AudioDeviceRole::Communications and capture device works fine all the time.
For Windows 10 capture device works all the time, no matter if you open it with AudioDeviceRole::Communications or not.
I've had the same problem. It seems like you can either use only AudioCategory_Other or create an instance of VoipPhoneCall and use only AudioCategory_Communications.
So the solution in my case was to use AudioCategory_Communications and create an outgoing VoipPhoneCall. You should implement the background agents as in Chatterbox VoIP sample app for the VoipCallCoordinator to work .
We are using a Cumulus server as our RTMFP server to implement a voice chat.
The voice streaming with NetStreams works perfectly fine almost no delay here, but we also want to transfer the activity level of the microphone of each member of the NetGroup.
So we tried transferring the data using NetGroup.post(data). That worked well, but had a delay of ~500ms even when testing with multiple clients on the same machine!
Obviously, half a second in microphone activity is just waaay to much delay to display it in any way.
Now we are trying to use direct routing with NetGroup.sendToAllNeighbors(, but it simply does nothing. I have read all the documentation on it and normally a NetGroup status event with "NetGroup.SendTo.Notify" should be triggered on the receiving clients. Instead, nothing happens.
Here is the code, which is called each frame:
var tsObject :TimestampedObject = new TimestampedObject();
tsObject.timestamp = (new Date()).getTime();
tsObject.sender = _netConnection.nearID;
tsObject.object = _mic.activityLevel;
_netGroup.sendToAllNeighbors(tsObject);
I know each frame is a bit much, but for now it is just a test case.
What are we doing wrong? As I said, voice communication itself works fine. As does Netgroup.post(data), which is just way too slow to use for this use case.