I have a desktop application which streams raw PCM data to my browser over a websocket connection. The stream looks like this ...\\x00\\x00\\x02\\x00\\x01\\x00\\x00\\x00\\x01\\x00\\xff\\xff\\xff\\xff\\....
The question is simple: can I play such a stream in HTML with the Web Audio API / WebRTC / ...?
Any suggestions are very welcome!
code edit
This code plays noise, randomly generated:
function myPCMSource() {
return Math.random() * 2 - 3;
}
var audioContext;
try {
window.AudioContext = window.AudioContext || window.webkitAudioContext;
audioContext = new AudioContext();
} catch(e) {
alert('Web Audio API is not supported in this browser');
}
var bufferSize = 4096;
var myPCMProcessingNode = audioContext.createScriptProcessor(bufferSize, 1, 1);
myPCMProcessingNode.onaudioprocess = function(e) {
var output = e.outputBuffer.getChannelData(0);
for (var i = 0; i < bufferSize; i++) {
output[i] = myPCMSource();
}
}
So changing the myPCMSource() to the websocket stream input, should make it work somehow. But it doesn't. I don't get any errors, but the API is not playing any sound nor noise.
Use a ScriptProcessorNode, but be aware that if there is too much load on the main thread (the thread that runs your javascript, draws the screen, etc.), it will glitch.
Also, your PCM stream is probably in int16, and the Web Audio API works in terms of float32. Convert it like so:
output_float[i] = (input_int16[i] / 32767);
that is, go from a [0; 65535] range to a [-1.0; 1.0] range.
EDIT
I was using output_float[i] = (input_int16[i] / 32767 - 1);, this article shows that you should use output_float[i] = (input_int16[i] / 32767);. Now it's working fine!
Just for the record, the ScriptProcessorNode is deprecated. See the MDN article for details. The feature was replaced by AudioWorklets and the AudioWorkletNode interface.
In short, a ScriptProcessorNode runs outside of the browser's internal audio thread, which creates at least on frame (128 samples) of latency. Worse, the ScriptProcessorNode often fails to respond quickly enough, if the thread is busy, so will just randomly drop the ball every so often.
Worklets are basically task-specific workers that run in one of the browsers internal threads (paint, layout, audio etc). Audio worklets run in the audio thread, and implement the guts of custom audio nodes, which are then exposed through the WebAudio API as normal.
Note: You are also able to run WebAssembly inside worklets to handle the processing.
The solution provided above is still useful, as the basic idea holds, but it would ideally use an audio worklet.
Related
I created a batch of sounds assembled with this tool:
AudioSprite
https://github.com/tonistiigi/audiosprite
The output is generally used for JS libraries, such as Howler, Zynga Jukebox, or SoundJS - but I wanted to see if it's possible to implement in AS3.
I started creating a Sound player that can load, parse and play the sounds based on the JSON and MP3 file this tool generates.
So far so good! ... except for loops.
Now, the big question is - is there a way to play a Sound-loop seamlessly given that all music & sounds coexist in the same MP3 file, and it has a start & end range to play and stop it?
Example of how the sounds are placed in the file:
mygame_sounds.mp3 = [BUZZ + LASER + BOING ... + TRACKLOOP]
I'm looking for a solution that does not involve using the SAMPLE_DATA Event (given it eats up a lot of CPU usage). If there's no way around it, please explain why.
So far I've had mild success using flash.utils.Timer objects triggered after a given AudioSprite's duration, but it's not consistent.
To stop / dispose of a non-looping sound, I rely on a Master Timer (running at very short intervals) and that seems to "cut" the sample appropriately. But I already tried using this Master Timer to play a looped-sound over and over - same latency issues.
Is there any method to predict / measure how much latency is to be expected by the time the sound completes one pass?
In SoundJS we could not find a way to allow smooth looping of audio sprites in AS3 and went with a timer. We found Web Audio was the only api that allowed smooth looping, and therefore recommended staying away from audiosprites for sounds that needed to loop smoothly if any other plugin might be used.
Hope that helps.
The reason of why you can't get smooth loops of a track retrieved from a larger audio file is that you cannot check sound position faster than once per SWF frame, which length depends on stage.frameRate and total processing time of your application and is generally varied. So, if your looping sounds lasts say 5.123 seconds (I don't care how many samples, just that its length does not make a full number of frames regardless of stage.frameRate), your sound will attempt to play for either 5.125 seconds (205 frames at 40 fps, IMO best bet for this particular sound), 5.133 seconds (154 frames at 30 fps) or some weird number of frames if the SWF would experience lag. The excess milliseconds cannot be totally controlled by any means due to AS3/Flash engine optimization. So, consider not using audio sprites and shift into audio packs (several audio files in an SWF, or one sound in an MP3).
Although I'm still working on the perfect solution, this is the best I could come up with:
Load the JSON file / ByteArray.
Parse the JSON file to obtain each sprites' ID, start and end times.
Load the MP3 file / ByteArray (requires loadCompressedDataFromByteArray()) into a master Sound object.
Once loaded, check if any sprites are marked as "loops".
Create separate Sound objects for the above loops, and extract the portion from the master Sound via loadPCMFromByteArray() with some "magic-numbers" (details below).
To play a one-shot sound, call the master Sound's play(sprite.start * 1000) (depending on the format, usually the JSON's start values are in seconds, needs to be in milliseconds).
To play a seamless-loop sound, call the individual Sound object's (created in step #5) play(0, 9999) method.
I won't go too deep in details on how to stop the sounds (SoundChannel.stop(), bam!), but I'll explain the "magic-numbers" mentioned above. See this snippet:
var goldenOffset:UInt = (64 << 5);
var goldenDuration:UInt = (64 << 2);
var sampleRate:UInt = 44100;
for (id in loops) {
var sprite:AudioSpriteItem = _mapSprites.get(id);
var loop:Sound = _mapLoops.get(id);
var sampleBytes = new ByteArray();
var samplesTotal:UInt = cast(sprite.duration * sampleRate + goldenDuration);
var samplesStart:UInt = cast(sprite.start * sampleRate + goldenOffset);
_sound.extract(sampleBytes, samplesTotal, samplesStart);
sampleBytes.position = 0;
loop.loadPCMFromByteArray(sampleBytes, samplesTotal, "float", true);
}
Quite honestly, these magic goldenOffset and goldenDuration values were just found via Trial-and-Error. I could get close to a seamless loop without them by just calculating the start and duration with the sampleRate (assuming 44100 by default), but each endings had a bit of a hiccup to it.
After several adjustments, those couple "64 left bit-shifted" values made the loops sound smoother.
I posted the Haxe project on github (compiled SWC also available in /bin folder) if you wish to try it / read through the code.
FLAudioSprite
Github page: https://github.com/bigp/FLAudioSprite
SWF Demo (Download): bit.ly/FLAudioSpriteSWFDemo
I am looking to take the audio input from the browser and stream it to multiple listeners. The intended use is for music, so the quality must mp3 standard or thereabouts.
I have attempted two ways, both yielding unsuccessful results:
WebRTC
Streaming audio directly between browsers works fine, but the audio quality seems to be non-customisable though what I have seen. (I have seen that it is using the Opus audio codec, but seems to not expose any controls).
Does anyone have any insight into how to increase the audio quality in WebRTC streams?
Websockets
The issue is the transportation from the browser to the server. The PCM audio data I can acquiring via the method below has proven too large to repeatedly stream to the server via websockets. The stream works perfectly in high speed internet environments, but on slower wifi it is un-usable.
var context = new webkitAudioContext()
navigator.webkitGetUserMedia({audio:true}, gotStream)
function gotStream (stream)
{
var source = context.createMediaStreamSource(stream)
var proc = context.createScriptProcessor(2048, 2, 2)
source.connect(proc)
proc.connect(context.destination)
proc.onaudioprocess = function(event)
{
var audio_data = event.inputBuffer.getChannelData(0)|| new Float32Array(2048)
console.log(audio_data)
// send audio_data to server
}
}
So the main question is, is there any way to compress the PCM data in order to make it easier to stream to the server? Or perhaps there is an easier way to go about this?
There are lots of ways to compress PCM data, sure, but realistically, your best bet is to get WebRTC to work properly. WebRTC is designed to do this - adaptively stream media - although you don't define what you mean by "multiple" listeners (there's a huge difference between 3 listeners and 300,000 simultaneous listeners).
There are several possible ways of resampling and/or compressing your data, none of them native though. I resampled the data to 8Khz Mono (your mileage may vary) with the xaudio.js lib from the speex.js environment. You could also compress the stream using speex, though that is used usually for audio only. In your case, I would probably send the stream to a server, compress it there and stream it to your audience. I really don't believe a simple browser to be good enough to serve data to a huge audience.
WebRTC seems to default to one mono channel around 42 kb/s, it seems to be primarily designed for voice.
You can disable the audio processing features using constraints to get a more consistent input from the browser using:
navigator.mediaDevices.getUserMedia({
audio: {
autoGainControl: false,
channelCount: 2,
echoCancellation: false,
latency: 0,
noiseSuppression: false,
sampleRate: 48000,
sampleSize: 16,
volume: 1.0
}
});
Then you also should set stereo and maxaveragebitrate params on the SDP:
let answer = await peer.conn.createAnswer(offerOptions);
answer.sdp = answer.sdp.replace('useinbandfec=1', 'useinbandfec=1; stereo=1; maxaveragebitrate=510000');
await peer.conn.setLocalDescription(answer);
This should output a string which looks like this:
a=fmtp:111 minptime=10;useinbandfec=1; stereo=1; maxaveragebitrate=510000
This could increase the bitrate up to 520kb/s for stereo, which is 260kps per channel. Actual bitrate depends on the speed of your network and strength of your signal tho.
I'm trying to get sound working on my iPhone game using the Web Audio API. The problem is that this app is entirely client side. I want to store my mp3s in a local folder (and without being user input driven) so I can't use XMLHttpRequest to read the data. I was looking into using FileSystem but Safari doesn't support it.
Is there any alternative?
Edit: Thanks for the below responses. Unfortunately the Audio API is horribly slow for games. I had this working and the latency just makes the user experience unacceptable. To clarify, what I need is sounething like -
var request = new XMLHttpRequest();
request.open('GET', 'file:///./../sounds/beep-1.mp3', true);
request.responseType = 'arraybuffer';
request.onload = function() {
context.decodeAudioData(request.response, function(buffer) {
dogBarkingBuffer = buffer;
}, onError);
}
request.send();
But this gives me the errors -
XMLHttpRequest cannot load file:///sounds/beep-1.mp3. Cross origin requests are only supported for HTTP.
Uncaught Error: NETWORK_ERR: XMLHttpRequest Exception 101
I understand the security risks with reading local files but surely within your own domain should be ok?
I had the same problem and I found this very simple solution.
audio_file.onchange = function(){
var files = this.files;
var file = URL.createObjectURL(files[0]);
audio_player.src = file;
audio_player.play();
};
<input id="audio_file" type="file" accept="audio/*" />
<audio id="audio_player" />
You can test here:
http://jsfiddle.net/Tv8Cm/
Ok, it's taken me two days of prototyping different solutions and I've finally figured out how I can do this without storing my resources on a server. There's a few blogs that detail this but I couldn't find the full solution in one place so I'm adding it here. This may be considered a bit hacky by seasoned programmers but it's the only way I can see this working, so if anyone has a more elegent solution I'd love to hear it.
The solution was to store my sound files as a Base64 encoded string. The sound files are relatively small (less than 30kb) so I'm hoping performance won't be too much of an issue. Note that I put 'xxx' in front of some of the hyperlinks as my n00b status means I can't post more than two links.
Step 1: create Base 64 sound font
First I need to convert my mp3 to a Base64 encoded string and store it as JSON. I found a website that does this conversion for me here - xxxhttp://www.mobilefish.com/services/base64/base64.php
You may need to remove return characters using a text editor but for anyone that needs an example I found some piano tones here - xxxhttps://raw.github.com/mudcube/MIDI.js/master/soundfont/acoustic_grand_piano-mp3.js
Note that in order to work with my example you're need to remove the header part data:audio/mpeg;base64,
Step 2: decode sound font to ArrayBuffer
You could implement this yourself but I found an API that does this perfectly (why re-invent the wheel, right?) - https://github.com/danguer/blog-examples/blob/master/js/base64-binary.js
Resource taken from - here
Step 3: Adding the rest of the code
Fairly straightforward
var cNote = acoustic_grand_piano.C2;
var byteArray = Base64Binary.decodeArrayBuffer(cNote);
var context = new webkitAudioContext();
context.decodeAudioData(byteArray, function(buffer) {
var source = context.createBufferSource(); // creates a sound source
source.buffer = buffer;
source.connect(context.destination); // connect the source to the context's destination (the speakers)
source.noteOn(0);
}, function(err) { console.log("err(decodeAudioData): "+err); });
And that's it! I have this working through my desktop version of Chrome and also running on mobile Safari (iOS 6 only of course as Web Audio is not supported in older versions). It takes a couple of seconds to load on mobile Safari (Vs less than 1 second on desktop Chrome) but this might be due to the fact that it spends time downloading the sound fonts. It might also be the fact that iOS prevents any sound playing until a user interaction event has occured. I need to do more work looking at how it performs.
Hope this saves someone else the grief I went through.
Because ios apps are sandboxed, the web view (basically safari wrapped in phonegap) allows you to store your mp3 file locally. I.e, there is no "cross domain" security issue.
This is as of ios6 as previous ios versions didn't support web audio api
Use HTML5 Audio tag for playing audio file in browser.
Ajax request works with http protocol so when you try to get audio file using file://, browser mark this request as cross domain request. Set following code in request header -
header('Access-Control-Allow-Origin: *');
I'm working on a local application ( it's not a website or nothing related ) and I have various FLVs with a very simple encryptation method by now (just like adding 10 at each byte).
I can load/play them using NetStream.appendBytes() after decrypting, but that happens only after I read all video data it's not streamed.
What I really need is to stream those videos from a remote url, and decrypting while receiving data, using a OSMF based player that I already have built.
I'm lost on how OSMF deals with FLV, otherwise, I would try to create a plugin or something like.
I'd be very thankful if someone point me how to deal with that.
But I'd be happy if someone help me to find a way to load a local file using OSMF, passing a ByteArray value, instead of a url (below). Or even giving me directions to create a OSMF plugin to solve my problem.
videoElement.resource = "video_url/video.flv";
This is my current code just to play my decoded FLV byte array
private function playBytes(bytes:ByteArray):void
{
// detecting it's header
if (bytes.readUTFBytes(3) != "FLV")
{
_text.appendText("\nFile \""+ file +"\" is not a FLV")
return void;
}
bytes.position = 0;
netConnection.connect(null);
netStream = new NetStream(netConnection);
netStream.client = { onMetaData:function(obj:Object):void { } }
video.attachNetStream(netStream);
addChild(video);
// put the NetStream class into Data Generation mode
netStream.play(null);
// before appending new bytes, reset the position to the beginning
netStream.appendBytesAction(NetStreamAppendBytesAction.RESET_BEGIN);
// append the FLV video bytes
netStream.appendBytes(bytes);
}
Interesting post, I'd be interested to see the answer. Looking at something similar myself, though not with a stream, I came across the following.
http://ntt.cc/2008/07/15/bitsreader-read-bits-from-given-bytearray.html
After passing the byte array you can use bits.read(8) of a 10 bit array. Perhaps this would send you down the correct path? Otherwise, I'm thinking you'd need to break it apart and essentially do smaller sections to buffer in order to concatenate all the buffered data...
Just a thought,
I'm building an online radio player using the AS3 code below:
private var soundChannel:SoundChannel;
private var stationUrl:String = "h t t p : / /205.188.215.230:8002/";
sound = new Sound();
sound.addEventListener(Event.ID3, onID3Change);
sound.load(new URLRequest(stationUrl));
soundChannel = sound.play();
private function onID3Change(e:Event):void
{
....
}
the sound plays successfully, but the problem is that the ID3 event is never triggered!
Does anyone know how to solve this?
ID3 doesn't exist in internet radio streams like this one. I am assuming you're talking about a SHOUTcast/IceCast stream.
For that, you need to implement the icy metadata protocol. For Flash, this is generally just done externally.
See this reference: http://www.smackfu.com/stuff/programming/shoutcast.html
Basically, you send icy-metadata: 1 in the headers of your GET request. The server then inserts metadata right into the middle of the stream, which you pull out before sending the data on to whatever is playing the stream. I'm not sure if this is even possible in Flash, but it certainly is possible to do this in PHP (or any server-side language really) and have your Flash application make a request to your PHP script to get that metadata.