I created a batch of sounds assembled with this tool:
AudioSprite
https://github.com/tonistiigi/audiosprite
The output is generally used for JS libraries, such as Howler, Zynga Jukebox, or SoundJS - but I wanted to see if it's possible to implement in AS3.
I started creating a Sound player that can load, parse and play the sounds based on the JSON and MP3 file this tool generates.
So far so good! ... except for loops.
Now, the big question is - is there a way to play a Sound-loop seamlessly given that all music & sounds coexist in the same MP3 file, and it has a start & end range to play and stop it?
Example of how the sounds are placed in the file:
mygame_sounds.mp3 = [BUZZ + LASER + BOING ... + TRACKLOOP]
I'm looking for a solution that does not involve using the SAMPLE_DATA Event (given it eats up a lot of CPU usage). If there's no way around it, please explain why.
So far I've had mild success using flash.utils.Timer objects triggered after a given AudioSprite's duration, but it's not consistent.
To stop / dispose of a non-looping sound, I rely on a Master Timer (running at very short intervals) and that seems to "cut" the sample appropriately. But I already tried using this Master Timer to play a looped-sound over and over - same latency issues.
Is there any method to predict / measure how much latency is to be expected by the time the sound completes one pass?
In SoundJS we could not find a way to allow smooth looping of audio sprites in AS3 and went with a timer. We found Web Audio was the only api that allowed smooth looping, and therefore recommended staying away from audiosprites for sounds that needed to loop smoothly if any other plugin might be used.
Hope that helps.
The reason of why you can't get smooth loops of a track retrieved from a larger audio file is that you cannot check sound position faster than once per SWF frame, which length depends on stage.frameRate and total processing time of your application and is generally varied. So, if your looping sounds lasts say 5.123 seconds (I don't care how many samples, just that its length does not make a full number of frames regardless of stage.frameRate), your sound will attempt to play for either 5.125 seconds (205 frames at 40 fps, IMO best bet for this particular sound), 5.133 seconds (154 frames at 30 fps) or some weird number of frames if the SWF would experience lag. The excess milliseconds cannot be totally controlled by any means due to AS3/Flash engine optimization. So, consider not using audio sprites and shift into audio packs (several audio files in an SWF, or one sound in an MP3).
Although I'm still working on the perfect solution, this is the best I could come up with:
Load the JSON file / ByteArray.
Parse the JSON file to obtain each sprites' ID, start and end times.
Load the MP3 file / ByteArray (requires loadCompressedDataFromByteArray()) into a master Sound object.
Once loaded, check if any sprites are marked as "loops".
Create separate Sound objects for the above loops, and extract the portion from the master Sound via loadPCMFromByteArray() with some "magic-numbers" (details below).
To play a one-shot sound, call the master Sound's play(sprite.start * 1000) (depending on the format, usually the JSON's start values are in seconds, needs to be in milliseconds).
To play a seamless-loop sound, call the individual Sound object's (created in step #5) play(0, 9999) method.
I won't go too deep in details on how to stop the sounds (SoundChannel.stop(), bam!), but I'll explain the "magic-numbers" mentioned above. See this snippet:
var goldenOffset:UInt = (64 << 5);
var goldenDuration:UInt = (64 << 2);
var sampleRate:UInt = 44100;
for (id in loops) {
var sprite:AudioSpriteItem = _mapSprites.get(id);
var loop:Sound = _mapLoops.get(id);
var sampleBytes = new ByteArray();
var samplesTotal:UInt = cast(sprite.duration * sampleRate + goldenDuration);
var samplesStart:UInt = cast(sprite.start * sampleRate + goldenOffset);
_sound.extract(sampleBytes, samplesTotal, samplesStart);
sampleBytes.position = 0;
loop.loadPCMFromByteArray(sampleBytes, samplesTotal, "float", true);
}
Quite honestly, these magic goldenOffset and goldenDuration values were just found via Trial-and-Error. I could get close to a seamless loop without them by just calculating the start and duration with the sampleRate (assuming 44100 by default), but each endings had a bit of a hiccup to it.
After several adjustments, those couple "64 left bit-shifted" values made the loops sound smoother.
I posted the Haxe project on github (compiled SWC also available in /bin folder) if you wish to try it / read through the code.
FLAudioSprite
Github page: https://github.com/bigp/FLAudioSprite
SWF Demo (Download): bit.ly/FLAudioSpriteSWFDemo
Related
----- UPDATE HAS BEEN ADDED BELOW -----
I have an issue with MediaRecorder API (https://www.w3.org/TR/mediastream-recording/#mediarecorder-api).
I'm using it to record the speech from the web page(Chrome was used in this case) and save it as chunks.
I need to be able to play it while and after it is recorded, so it's important to keep those chunks.
Here is the code which is recording data:
navigator.mediaDevices.getUserMedia({ audio: true, video: false }).then(function(stream) {
recorder = new MediaRecorder(stream, { mimeType: 'audio/webm; codecs="opus"' })
recorder.ondataavailable = function(e) {
// Read blob from `e.data`, decode64 and send to sever;
}
recorder.start(1000)
})
The issue is that the WebM file which I get when I concatenate all the parts is corrupted(rarely)!. I can play it as WebM, but when I try to convert it(ffmpeg) to something else, it gives me a file with shifted timings.
For example. I'm trying to convert a file which has duration 00:36:27.78 to wav, but I get a file with duration 00:36:26.04, which is 1.74s less.
At the beginning of file - the audio is the same, but after about 10min WebM file plays with a small delay.
After some research, I found out that it also does not play correctly with the browser's MediaSource API, which I use for playing the chunks. I tried 2 ways of playing those chunks:
In a case when I just merge all the parts into a single blob - it works fine.
In case when I add them via the sourceBuffer object, it has some gaps (i can see them by inspecting buffered property).
697.196 - 697.528 (~330ms)
996.198 - 996.754 (~550ms)
1597.16 - 1597.531 (~370ms)
1896.893 - 1897.183 (~290ms)
Those gaps are 1.55s in total and they are exactly in the places where the desync between wav & webm files start. Unfortunately, the file where it is reproducible cannot be shared because it's customer's private data and I was not able to reproduce such issue on different media yet.
What can be the cause for such an issue?
----- UPDATE -----
I was able to reproduce the issue on https://jsfiddle.net/96uj34nf/4/
In order to see the problem, click on the "Print buffer zones" button and it will display time ranges. You can see that there are two gaps:
0 - 136.349, 141.388 - 195.439, 197.57 - 198.589
136.349 - 141.388
195.439 - 197.57
So, as you can see there are 5 and 2 second gaps. Would be happy if someone could shed some light on why it is happening or how to avoid this issue.
Thank you
It's 7 months later so I guess you resolved this, but in case not...
When we started working with the MediaRecorder we were having a few issues, including recordings disappearing (Maybe going over a RAM quota and then the arrays were deallocated or something like that)
What solved all our issues was to immediately put each chunk into an indexdb objectStore so it is saved to disk, and at the end of the recording, build all those chunks into a blob and download. No further working with the chunks, only the complete file.
I know this doesn't answer your question but maybe it helps.
I have an AS3 music player built into an app that I'm putting together. It works perfectly with almost every file I've used, but there is one file that it stops early on. The file is roughly 56 seconds long, the player stops at about 44 seconds. I'm using trace to show the length, and for every other song the length is correct. In this case, trace shows roughly 44 seconds instead of 56. Here's the code I use to load the file:
length = 0;
request = new URLRequest(fileAddress);
track = new Sound();
track.load(request);
track.addEventListener(Event.COMPLETE, TrackLoaded);
And here's the TrackLoaded function:
private function TrackLoaded(e:Event):void{
length = track.length;
if (playWhenLoaded == true){
trackChannel = track.play(0);
trackChannel.addEventListener(Event.SOUND_COMPLETE, TrackFinishedPlaying);
playWhenLoaded = false;
}
Works perfectly with every other file. What am I missing?
Are you willing to host this 56 sec MP3 somewhere for download & analysis? Or if you can yourself check the header info via a Hexeditor. I suspect either of two things:
1) Header has incorrect time length embedded and Flash takes that as final duration and stops there. After all why read anymore remaining bytes? they could be just metadata not audio samples Besides what encoder would lie about true duration? So its accepted as final duration even if your ears know that its incorrect.
2) MP3 samplerate /bitrate issue: Consider what sample rate is this one problem MP3? Check the sample rate against of a working MP3. Also are these various found sounds or you made each yourself? I ask to confirm you put the same settings for each file yet this one does not work?
In anycase I think you could solve this particular MP3 by re-encoding it. Maybe save as WAV or AIFF first then take that new uncompressed audio and convert back to a MP3 with samplerate of 44100 khz + Stereo sound + Constant Bitrate (avoid Variable B.Rate like hell if you dont want issues)
Checking and fixing either of the above should get you are correctly parsed MP3. Hope it helps
I am building a small Windows Phone 8 app (a Christian-Orthodox calendar) which has a background agent which should update the live tile. The app will require access to the contacts in the phone so I opted out of internet access so backend tile generation is, at least now out of question. I personally would not trust an app that has access to my contacts AND to internet.
Recently my scheduled agent (which generates three PNGs) started OutOfMemoryException-ing on me. Consistently. I've used DeviceStatus to query and debug its behavior.
It's hard to call this a memory leak since between all three tile generations if I call GC.Collect it won't throw OutOfMemoryException. If it were a true memory leak some (large and/or many) objects would remain referenced by other live/root objects and no amount of GC.Collect will help. In my case GC.Collect WILL help. I can continue using GC.Collect but I feel dirty doing so.
As I'm developing the app free and open-source you can view in detail all the code of the project at the current state of development at http://orthodoxcalendar.codeplex.com
The tile generation consists of taking a background and overlaying two other images on that background. Basically for each of the three PNGs generated I do
var bytes1 = (byte[])resourceManager.GetObject(resourceName1);
var stream1 = new MemoryStream(bytes);
var bytes2 = (byte[])resourceManager.GetObject(resourceName2);
var stream2 = new MemoryStream(bytes);
var bytes3 = (byte[])resourceManager.GetObject(resourceName3);
var stream3 = new MemoryStream(bytes);
var writeableBitmap1 = BitmapFactory.New(size.Width, size.Height).FromStream(stream1); // background
var writeableBitmap2 = BitmapFactory.New(size.Width, size.Height).FromStream(stream2); // first overlay
var writeableBitmap3 = BitmapFactory.New(size.Width, size.Height).FromStream(stream3); // second overlay
writeableBitmap1.Blit(new Point(0, 0), writeableBitmap2, new Rect(0, 0, width2, height2), Colors.White, BlendMode.Alpha);
writeableBitmap1.Blit(new Point(0, 0), writeableBitmap3, new Rect(0, 0, width3, height3), Colors.White, BlendMode.Alpha);
writeableBitmap1.DrawText("Some text", new Point(5, 139), Color.Black, 17);
writeableBitmap1.Invalidate(); // flatten things
using(var outputStream = new WhateverStream())
{
PNGWriter.Write(writeableBitmap1, outputStream);
}
writeableBitmap1.SetSource(new MemoryStream(MiscData.MinimumPng)); // set the writeable bitmap to a 1x1 transparent PNG to, hopefully, force it to release unamanaged memory or other stuff
writeableBitmap2.SetSource(new MemoryStream(MiscData.MinimumPng));
writeableBitmap3.SetSource(new MemoryStream(MiscData.MinimumPng));
stream1.Dispose();
stream2.Dispose();
stream3.Dispose();
The code, if you'll check out the project, is not exactly like above since I've wrapped almost all dependencies in adapters and extracted interfaces. Across many assemblies. The above code is the simplified version which just shows, what I consider to be, the relevant code lines.
A few explanations for the code above :
all this code is run in the background agent inside a Dispatcher.BeginInvoke since you can't seem to manipulate a WritableBitmap on any other thread than the UI thread
The PNG data is stored in another assembly as resx. I know this fattens the assembly but I need this to reuse it across platforms as the assembly is a PCL
Creating the WriteableBitmap directly using a byte array seems to fail in a mysterious way so I'm wrapping it in a MemoryStream and somehow, this way, it works
The PNG writer is taken from ToolStack.
It's not feasible to pre-generate the images since there are multiple versions of "first overlay", "second overlay" and, mostly the "Some text". It would mean tens of thousands of images, at least.
The heart of the question : Am I doing something awfully wrong that I'm not aware of? The only thing that pops in my mind is that JPEGs are generated faster and with less memory consumption but they won't have transparency which I desire. Can this be actually called a memory leak?
LATER EDIT : It seems that after some more debugging it changed its behavior from the one above to a true memory leak. I switched from PNG generation to JPEG generation and the memory is lower now. The input images are still PNG but at the other end a JPEG will be spit. The memory footprint went several megabytes below the previous threshold(s).
SECOND EDIT : I put the logic in a 10.000 repeat loop on a button and there doesn't seem too much memory consumption. I am beginning to think that there isn't really a memory leak but just higher memory consumption during the generation and that's enough to bring the fragile agent down.
In doing a similar thing I've had to explicitly set the writeablebitmaps to null (even though should be unnecessary) before calling GC.Collect.
Additionally, it may be better to create and destroy (and collect) each of the images in turn, rather than creating them all and then destroying them all. This will help with the overhead at any one point.
Also note that when tracking the memory use in the debugger, the debugger adds about 3mb of overhead that you won't see when live.
I'm working on a local application ( it's not a website or nothing related ) and I have various FLVs with a very simple encryptation method by now (just like adding 10 at each byte).
I can load/play them using NetStream.appendBytes() after decrypting, but that happens only after I read all video data it's not streamed.
What I really need is to stream those videos from a remote url, and decrypting while receiving data, using a OSMF based player that I already have built.
I'm lost on how OSMF deals with FLV, otherwise, I would try to create a plugin or something like.
I'd be very thankful if someone point me how to deal with that.
But I'd be happy if someone help me to find a way to load a local file using OSMF, passing a ByteArray value, instead of a url (below). Or even giving me directions to create a OSMF plugin to solve my problem.
videoElement.resource = "video_url/video.flv";
This is my current code just to play my decoded FLV byte array
private function playBytes(bytes:ByteArray):void
{
// detecting it's header
if (bytes.readUTFBytes(3) != "FLV")
{
_text.appendText("\nFile \""+ file +"\" is not a FLV")
return void;
}
bytes.position = 0;
netConnection.connect(null);
netStream = new NetStream(netConnection);
netStream.client = { onMetaData:function(obj:Object):void { } }
video.attachNetStream(netStream);
addChild(video);
// put the NetStream class into Data Generation mode
netStream.play(null);
// before appending new bytes, reset the position to the beginning
netStream.appendBytesAction(NetStreamAppendBytesAction.RESET_BEGIN);
// append the FLV video bytes
netStream.appendBytes(bytes);
}
Interesting post, I'd be interested to see the answer. Looking at something similar myself, though not with a stream, I came across the following.
http://ntt.cc/2008/07/15/bitsreader-read-bits-from-given-bytearray.html
After passing the byte array you can use bits.read(8) of a 10 bit array. Perhaps this would send you down the correct path? Otherwise, I'm thinking you'd need to break it apart and essentially do smaller sections to buffer in order to concatenate all the buffered data...
Just a thought,
I have an AS3 application that loads various SWFs at runtime. The loading animation that is being used has a fairly long in and out animation that I don't want to show if the target SWF is in the browser cache.
So at the moment each SWF is loaded in as required using Greensock's SWFLoader in a basic manner:
var context:LoaderContext = new LoaderContext();
context.applicationDomain = ApplicationDomain.currentDomain;
loader = new SWFLoader("mySWF.swf", {name:"sectionLoader",context:context,auditSize:true,onOpen:onLoadInit,onProgress:onLoadProgress, onComplete:onCompleteLoadHandler, onError:onLoadErrorHandler});
loader.load();
My goal is to do something before calling loader.load(); to determine if the load operation will require the request to go beyond the browser cache, but before I get into R&Ding something I thought I'd ask if anyone has already done something similar.
A few more thoughts I've had so far:
Just keeping track of what has been loaded in AS3 isn't good enough because if the user clears their cache they might be left loading a large SWF on a slow connection with no indicator.
Might a combination of LoaderItem.httpStatus and LoaderItem.auditSize() be worth investigating?
Is there a better loading framework for AS3 that I should be looking into instead of the Greensock classes?
Ideally I would prefer to also have some kind of version detection to span sessions that could be months apart, but one step at a time.
when you are doing any HTTP request, the responce comes up with HTTPStatus property. In AS3 you just need to chek if
HttpStatusEvent.status == 304
And for httpStatus in greensock library.
Basically 304 code means that no chages has been made on server side to the resource which user has requested. Which eventually leads to conclution that the resource is in the cache.
UPDATE
If this will not fit your needs try storing some variable for should you play the animation or not in Cookies or in Session variables.