I'm currently programming a little game which involved that a dynamically created music is playing, and on a specific tone, a function is called to update the game.
I have no problem with dynamically creating music, either using SampleDataEvent.SAMPLE_DATA or the wonderful standingwave2 lib, but I can't find out how to synchronize sound with code.
I know the "sync" note play every X ms (let's say 500), so I've tried to start a timer which ticks every 500ms right after starting the sound, but it gets eventually out of sync. I'm not sure if the timer isn't good enough to follow the path
I know there's a way to put music on Adobe IDE Frames, play sound as "stream" and then put some code on each frame so I can know where it's called, but I can't dynamically create music that way.
So, does anyone knows a way to synchronize my function call with the sound I'm creating ?
I think this depends on when and how does the music generation takes place. If you are generating that music prior to running the game, then you can yield time offset list in that music when the particular tone is generated into that music, then you make a sorted array out of those values, then when the music is actively started, you take flash.utils.getTimer() value and store it as your base time. After this, each frame you check if current getTimer() value is greater than current array position, and if so, the function you want is called, and you advance one position in the array, to be ready for the next pre-set occurrence of your desired tone.
If, on the other hand, you generate music on the fly, a couple of frames length each, then you have to lock getTimer() value at the start of the game and (supposedly) music generation, so that each pair of values you put into sampleData are exactly 1/44100 second of music played. You then count those pairs (on the fly, of course) until it'll be time to insert your desired tone into the generated music, then you'll have an offset from sound start. Convert it to milliseconds, then check each frame if current getTimer() minus stored tick count is greater or equal to discovered offset, and if true, call the function.
As I know sounds playing correlates with frames even if you add them dynamicaly. Try to use Event.ENTER_FRAME. If you know framerate (by default it's equal to 24 fps) and delay (X ms; when a "sync" note plays) then you can get a "sync" frame's index: index = fps * delay. Because for syncronization of a sound and frames important only a nominal fps, not real. Count frames in the Event.ENTER_FRAME handler. When you will achieve the "sync" frame then you can execute your code.
Why don't you just inform your mechanics code about what happens with music when you do render the music handling SampleDataEvent.SAMPLE_DATA? It should be pretty accurate, and you'll never be out of sync for more than one sound chunk (which is usually 2048-4096 float pairs as far as I remember, means 2048/44100 - 4096/44100 ~= 1/22 - 1/11 seconds). Also, I haven't checked this, but I believe, SAMPLE_DATA is fired right after the existing sound chunk started to play, so you can have the next one after it finishes ready, which would mean, if you write down the time of sound render start, then the actual sound will be played the exact that time later(if your system isn't overloaded), so you can calculate it very precisely, down to milliseconds.
You need to enapsulate that WAV generated music within a FLV stream (using only audio tags). Between the FLV audio tags insert FLV metadata tags, which you will receive through onMetaData just when that portion is playing. Since you are generating music on the fly, you might wanna use NetStream.appendBytes() instead of just passing WAV files to NetStream.
You need to familiarize yourself with how FLV works, how appendBytes works and how to create a FLV (which you write into appendBytes() as if you are writing it to a file) which contains WAV audio.
Related
I'm trying to make a program that can convert ORG files into WAV files directly. The ORG format is similar to MIDI, in the sense that it is a list of "instructions" about when and how to play specific instruments, and a program plays these instruments for it to create the song.
However, as I said, I want to generate a WAV directly, instead of just playing the ORG. So, in a sense, I want to "play" the sounds into a WAV. I do know the WAV format and have created some files from raw PCM samples, but this isn't as simple.
The sounds generated by the ORG come from a bunch of files containing WAV samples I have. They're mono, 8-bit samples should be played at 22050Hz. They're all under a second long, and the largest aren't more than 11KB. I would assume that to play them all after each other, I would simply put the samples into the WAV one after the other. It isn't that simple though, as the ORG can have up to 16 different instruments playing at once, and each note of each instrument also has a pan (i.e. a balance, allowing stereo sound). What's more, each ORG has its own tempo (i.e. milliseconds between each point a sound can be played), and some sounds may be longer than this tempo, which means that two sounds on the same instrument can overlap. For instance, a note plays on an instrument, 90 milliseconds later the same note plays on the same instrument, but the first not hasn't finished, hence the first note plays into the second.
I just thought to explain all of that to be sure the situation is clear. In any case, I'd basically like to know how I would go about converting or "playing" an ORG (or if you like, a MIDI (since they're essentially the same)) into a WAV. As I mentioned each note does have a pan/balance, so the WAV would also need to be stereo.
If it matters at all, I'll be doing this in ActionScript 3.0 in FlashDevelop. I don't need any code (as that would be asking someone to do the work for me), but I just want to know how I would go about doing this correctly. An algorithm or two may be handy as well.
First let me say AS3 is not the best language to do these kind of things. Super collider would be a better and easier choice.
But if you want to do it in AS3 here's a general approach. I haven't tested any of it, this is pure theory.
First, put all your sounds into an array, and then find a way of matching the notes from your midi file to a position in the array.
I don't know the format of midi in depth, but I know the smallest value is a tick, and the length of a tick depends on the BPM. Here's the formula to calculate a midi tick: Midi Ticks to Actual PlayBack Seconds !!! ( Midi Music)
Let's say your tick is 2ms in length. So now you have a base value. You can fill a Vector (like an Array but faster) with what happens at every tick. If nothing happens at a particular tick, then insert a null value.
Now the big problem is reading that Vector. It's a problem because the Timer class does not work at small values like 2ms. But what you can do is check the ellapsed time in ms since the app started using getTimer(). You can have some loop that will check the ellapsed time, and whenever you have 2ms more, you read the next index in the Vector. If there are notes on that index, you play the sounds. If not you wait for the next tick.
The problem with this, is that if a loop goes on for more than 15 seconds (I'm not sure of that value) Flash will think the program is not responding and will kill it. So you have to take care of that too, ending the loop and opening a new one before Flash kills your program.
Ok, so now you have sounds playing. You can record the sounds that flash is making (wavs, mp3, mic) with a library called Standing Wave 3.
https://github.com/maxl0rd/standingwave3
This is very theoretical... and I'm quite sure depending on the number of sounds you want to play you can freeze your program... but I hope it will help to get you going.
As far as I've read in the manuals, when you have to play a sound, you need a Sound object, and make a temporary SoundChannel object that will control the actual playback. I want to know if there is a memory-efficient way of managing those SoundChannel objects. So far it seems that these are of "fire-and-forget" type of objects, and the only way to make them semi-persistent is make the call to Sound.play() with a really great number of plays. But this approach will not work for one-time sounds, like an arrow shot or a button click, for example. And if I call SoundChannel.stop() I can as well discard the object as there is no means to make it resume playing. Is there any solution to not to spawn SoundChannel objects like crazy, and to be able to handle both one-time sound plays and infinite-time sounds aka background music?
SoundChannel is indeed meant to be throw-away, and this kind of heap usage comes with the territory of using a language like ActionScript.
You shouldn't worry about GC usage from sounds - premature optimization is evil! The best you can do is just reuse your Sound object instead of creating a new one each play. There shouldn't be much of a GC issue if you are playing a reasonable number of sounds per frame, say, in a game. SoundChannels are lightweight and reference a single copy of the audio data, so they aren't such a big deal. There will probably be much heavier allocations to worry about, such as game objects or bitmaps.
You could avoid using SoundChannel by dynamically mixing the audio using SampleDataEvent, but this will certainly have the opposite effect and be much more processor-intensive, not to mention more difficult to code.
If you are really worried about the GC, you could use System.pauseForGCIfCollectionImminent method to hint the GC to run during a non-intrusive time, such as during a transition in a game.
I'm new to Actionscript. There's probably a better way to do this, and if there is, I'm all ears.
What I'm trying to do is have a background layer run for, say 150 seconds in a loop. Then have another layer (we'll call it Layer 1) with a single object on it loop for 50 seconds. Is there a way to have Layer 1 loop 3 times inside of that 150 seconds that the background layer is looping?
Here's the reason I want Layer 1 to be shorter:
When a certain combination is entered (for example, A1), an item will pop out of and in front of the object on Layer 1.
I haven't written any code for it yet, but my hopeful plan is to have the background layer run continuously then have different scene sections on Layer 1 for each of the items coming out of the object on Layer 1. That way when A1 is entered, Layer1 can goToAndPlay(51) without messing up the background layer.
If it helps you understand it at all, it's a vending machine project. My group's vending machine is the TARDIS. The TARDIS is flying through space while you're entering what you want out of the vending machine and stuff is popping out of it.
If I understand correctly, the background is a MovieClip that loops within its own timeline. When Flash plays through a timeline, the timing is dependent on the performance of the computer and how complex the animation is. You can add an audio track set to 'streaming' to lock the timing down, which will then drop frames if the CPU is overloaded. I have used a silent sound set to loop infinitely and play mode 'streaming' to do this when there is no audio to be used.
Instead of using timeline animations I would recommend using TweenMax http://www.greensock.com/tweenmax/ as it allows tween chaining, that is creating a chain of sequential and parallel tweens. When you use a tween you define the timing in seconds and can use values like 1.25 seconds. It will be accurate to the timing you define. You can also run methods on complete, use easing and all sorts of goodies. If you get comfortable using this you will be able to create much more complex interactions in your Flash projects and also be able to change animations and timing much easier than messing with the timeline.
In fact when hiring Flash developers we always screen candidates by asking if they prefer to do animations on the timeline or programmatically. Although Flash is on its way out, still good to learn as the ideas will apply to javascript and whatever new technology comes about.
I'm writing an app that requires an audio stream to be recording while a backing track is played. I have this working, but there is an inconsistent gap in between playback and record starting.
I don't know if I can do anything to make the sync perfect every time, so I've been trying to track what time each stream starts so I can calculate the delay and trim it server-side. This also has proved to be a challenge as no events seem to be sent when a connection starts (as far as I know). I've tried using various properties like the streams' buffer sizes, etc.
I'm thinking now that as my recorded audio is only mono, I may be able to put some kind of 'control signal' on the second stereo track which I could use to determine exactly when a sound starts recording (or stick the whole backing track in that channel so I can sync them that way). This leaves me with the new problem of properly injecting this sound into the NetStream.
If anyone has any idea whether or not any of these ideas will work, how to execute them, or some alternatives, that would be extremely helpful! Been working on this issue for awhile
The only thing that comes to mind is to try and use metadata, flash media streams support metadata and the onMetaData callback. I assume you're using flash media server for the audio coming in and to record the audio going out. If you use the send method while your streaming the audio back to the server, you can put the listening audio track's playhead timestamp in it, so when you get the 2 streams back to the server you can mux them together properly. You can also try encoding the audio that is streamed to the client with metadata and try and use onMetaData to sync them up. I'm not sure how to do this, but a second approach is to try and combine the 2 streams together as the audio goes back so that you don't need to mux them later, or attach it to a blank video stream with 2 audio tracks...
If you're to inject something into the NetStream... As complex as SOUND... I guess here it would be better to go with Socket instead. You'll be directly reading bytes. It's possible there's a compression on the NetStream, so the data sent is not raw sound data - some class for decompressing the codec there would be needed. When you finally get the raw sound data, add the input in there, using Socket.readUnsignedByte() or Socket.readFloat(), and write back the modified data using Socket.writeByte(), or Socket.writeFloat().
This is the alternative with injecting the back into the audio.
For syncing, it is actually quite simple. Even though the data might not be sent instantly, one thing still stays the same - time. So, when user's audio is finished, just mix it without anything else to the back track - the time should stay the same.
IF the user has slow internet DOWNLOAD, so that his backtrack has unwanted breaks - check in the SWF if the data is buffered enough to add the next sound buffer (usually 4096 bytes if I remember correctly). If yes, continue streaming user's audio.
If not, do NOT stream, and start as soon as the data catches back on.
In my experience NetStream is one of the most inaccurate and dirty features of Flash (NetStream:play2 ?!!), which btw is quite ironic seeing how Flash's primary use is probably video playback.
Trying to sync it with anything else in a reliable way is very hard... events and statuses are not very straight forward, and there are multiple issues that can spoil your syncing.
Luckily however, netStream.time will tell you quite accurately the current playhead position, so you can eventually use that to determine starting time, delays, dropped frames, etc... Notice that determining the actual starting time is a bit tricky though. When you start loading a netStream, the time value is zero, but when it shows the first frame and is waiting for the buffer to fill (not playing yet) the time value is something like 0.027 (depends on the video), so you need to very carefully monitor this value to accurately determine events.
An alternative to using NetStream is embedding the video in a SWF file, which should make synchronization much easier (specially if you use frequent keyframes on encoding). But you will lose quality/filesize ratio (If I remember correctly you can only use FLV, not h264).
no events seem to be sent when a connection starts
sure there does.. NetStatusEvent.NET_STATUS fires for a multitude of reasons for NetConnections and Netstreams, you just have to add a listener and process the contents of NET_STATUS.info
the as3 reference docs here and you're looking for NET_STATUS.info
I'm building a remote presentation tool in AS3. In a nutshell, one user (the presenter) has access to a "table of contents" HTML page with links for each slide in the presentation, and an arbitrary number of viewers can watch the presentation on another page, which in turn is in the form of a SWF that polls the server every second to ensure that it's on the right slide. Whenever the admin clicks a slide link in the TOC, the database gets updated, and on its next request the presentation swf compares the label of the slide it's currently displaying to the response it got from the server. If the response differs from the current label, the swf scrubs through the timeline until it finds the right frame label; otherwise, it does nothing and waits for the next poll result (a second later).
Each slide consists of a movieclip with its own nested timeline that loops as long as the slide is displayed. There's no actionscript controlling any of the nested movieclips, nor is there any actionscript on the main timeline except the stop();s on each keyframe (each of which is a slide in the presentation).
Everything is built and working perfectly. The only thing that's troubling is that if the presentation swf is open for long enough (say, 20 minutes), the polling starts to have a noticeable effect on the framerate of the movieclips animating on any given slide. That is, every second, there's a noticeable drop in the framerate of the animations that lasts about three-tenths of a second, which is quite noticeable (and hence is a deal-breaker for the whole presentation suite!).
I know that AS3 has issues with memory management, and I've tried to be diligent in my re-use of objects and event listeners. The code itself is dead simple; there's a Timer instance that fires every second, which triggers a new URLRequest to be loaded by a URLLoader. The URLLoader is reused from call to call, while the URLRequest is not (it needs to be initialized with a new cache-killing value each time, retrieved from a call to new Date().time). The only objects instantiated in the entire class are the Timer, the URLLoader, the various URLRequests (which should be garbage-collected), and the only event listeners are on the Timer (added once), the URLLoader (added once), and on the routines that scrub backwards and forwards in the timeline to find the right slide (and they're removed once the correct slide is found).
I've been using mr doob's stats package to monitor memory usage, which definitely grows over time, so there's gotta be a leak somewhere (it grows from ~30 MB initially to > 200 MB after some scrubbing and about 25 minutes of uptime).
Does anyone have any ideas as to what might be causing the performance problems?
UPDATE: I'm not entirely sure the performance troubles are tied directly to memory; I ran an instance of the presentation swf for about 15 minutes and although memory usage only climbed to around 70 MB (and stayed there), a noticeable hiccup started appearing at one-second intervals, coinciding with the polling calls (tracked via Firebug's Net panel). What else might cause stuttering movieclips?
I know this is coming a bit late, but I have been using Flash Builder's profiler frequently and one thing I found is that the TimerEvent generated by the timer class
uses up quite a bit of memory individually and
seems to not get released properly during garbage collection (even if you stopped the timer and removed all references to it).
A new event is generated for each Timer tick. I use setInterval instead, even though a few AS3 evangelists seem to recommend against that. I don't know why. setInterval still generates timer events, but they appear to be garbage-collected properly over time.
So one strategy may be that
you replace the Timer with a call to setInterval() ... which is arguably more robust code anyway and
(CAUTION) force garbage collection on each slide scrub (but not on each poll). See this question for more details on the pros and cons.
The second suggestion is only a stop-gap measure. I really encourage you to use the profiling tools to find the leak. Flash Builder Pro has a 60-day trial that might help.
Finally, when moving to a completely new slide SWF (not a new timeline position in the current slide), how are you making sure that the previous slide SWF got unloaded properly? Or am I misunderstanding your setup and there is only one actual slide SWF?
Just two things that came into my mind:
Depending on the version of the Flash player and the cpu usage the garbage collections sometimes does not start before 250 MB (or even more) memory are consumed.
Moviesclips, Sprites, Loader and whatever that has an Eventlistener listening will not be killed by the garbage collection.
So I believe your problem is, that either the slides or the loader are not cleaned correctly after you used them, so the were keept in memory.
A good point to start reading: http://www.gskinner.com/blog/archives/2006/06/as3_resource_ma.html