need your advice.
I have a web-serivice which generates mp3 files out of wavs.
The process of conversion takes time and I want visitor to be able to start listening right away, while the conversion is still going on.
Having tried the html5 tag I found that I can only play the part of mp3 file which was ready at the moment the mp3 file was fetched. I mean, it doesn't seem to care that the file might have grown since it was fetched.
What is the right way to approach this situation?
Thanks in advance for any info.
I believe that you can use JPlayer to play them. One of the features is that it does not preload.
EDIT : The
HTML5 audio player also has the preload attribute which can have the following values:
"none": will not prefetch anything
"metadata" (not sure if it's the correct word): will get some basic stuff like length, sample rate
"auto": will prefetch the entire mp3
You need to get a bit more control over the serving process, rather than just leaving it up to your web server.
When your web server responds to an HTTP request, it includes a Length: header that tells the client how big the requested resource is, in bytes. Your web server will only send up to the length available at the time of the request, because it doesn't know the file is about to be appended to. The client will download all of that data, but from the client's perspective, it will have downloaded the entire file when really the file wasn't even done being encoded yet.
To work around this issue, you need to pipe the output of your encoder to both a file, and your client at the same time. For the response data to the client, do not include a Length: header at all. Most clients will work with chunked encoding, allowing you to be compliant with HTTP/1.1. Some clients (early Android, old browsers, old VLC) cannot handle chunked encoding, and will just stream the data as it comes in.
How you do this specifically depends entirely on what you're using server-side, which you didn't specify in your question. Personally, I use Node.js for this as it makes the process very easy. You can simply pipe to both streams. Be careful that if you use the multiple pipe method, the pipes only run as fast as the slowest. Some streaming clients (such as VLC) will lower the TCP window size to not allow too much data to have to be buffered client-side. When this occurs, your writes to disk will run at the speed of the client. This may not matter to you, but it is something to be aware of.
Related
I am trying to build a web application which will need to have audio streaming functionality implemented in some way. Just to give you guys some context: It is designed to be a purely auditive experience/game/idkhowtocallit with lots of different sound assets varying in length and thus file size. The sound assets to be provided will consist of ambient sounds, spoken bits of conversation, but also long music sets (up to a couple of hours). Why I think I won't be able to just host these audio files on some server or CDN and serve them from there is, because the sound assets will need to be fetched and played dynamically (depending on user interaction) and as instantly as possible.
Most importantly, consuming larger files (like the music sets and long ambient loops) as a whole doesn't seem to be client-friendly at all to me (used data consumption on mobile networks and client-side memory usage).
Also, without any buffering or streaming mechanism, the client won't be able to start playing these files before they are downloaded completely, right? Which would add the issue of high latencies.
I've tried to do some online research on how to properly implement a good infrastructure to stream bigger audio files to clients on the server side and found HLS and MPEG-DASH. I have some experience with consuming HLS players with web players and if I understand it correctly, I would use some sort of one-time transformation process (on or after file upload) to split up the files into chunks and create the playlist and then just serve these files via HTTP. From what I understand the process should be more or less the same for MPEG-DASH. My issue with these two techniques is that I couldn't really find any documentation on how to implement JavaScript/TypeScript clients (particularly using the Web Audio API) without reinventing the wheel. My best guess would be to use something like hls.js and bind the HLS streams to freshly created audio elements and use these elements to create AudioSources in my Web Audio Graph. How far off am I? I'm trying to get at least an idea of a best practice.
To sum up what I would really appreciate to get some clarity about:
Would HLS or MPEG-DASH really be the way to go or am I missing a more basic chunked file streaming mechanism with good libraries?
How - theoretically - would I go about limiting the amount of chunks downloaded in advance on the client side to save client-side resources, which is one of my biggest concerns?
I was looking into hosting services as well, but figured that most of them are specialized in hosting podcasts (fewer but very large files). Has anyone an opinion about whether I could use these services to host and stream possibly 1000s of files with sizes ranging from very small to rather large?
Thank you so much in advance to everyone who will be bothered with helping me out. Really appreciate it.
Why I think I won't be able to just host these audio files on some server or CDN and serve them from there is, because the sound assets will need to be fetched and played dynamically (depending on user interaction) and as instantly as possible.
Your long running ambient sounds can stream, using a normal HTMLAudioElement. When you play them, there may be a little lag time before they start since they have to begin streaming, but note that the browser will generally prefetch the metadata and maybe even the beginning of the media data.
For short sounds where latency is critical (like one-shot user interaction sound effects), load those into buffers with the Web Audio API for playback. You won't be able to stream them, but they'll play as instantly as you can get.
Most importantly, consuming larger files (like the music sets and long ambient loops) as a whole doesn't seem to be client-friendly at all to me (used data consumption on mobile networks and client-side memory usage).
If you want to play the audio, you naturally have to download that audio. You can't play something you haven't loaded in some way. If you use an audio element, you won't be downloading much more than what is being played. And, that downloading is mostly going to occur on-demand.
Also, without any buffering or streaming mechanism, the client won't be able to start playing these files before they are downloaded completely, right? Which would add the issue of high latencies.
If you use an audio element, the browser takes care of all the buffering and what not for you. You don't have to worry about it.
I've tried to do some online research on how to properly implement a good infrastructure to stream bigger audio files to clients on the server side and found HLS and MPEG-DASH.
If you're only streaming a single bitrate (which for audio is usually fine) and you're not streaming live content, then there's no point to HLS or DASH here.
Would HLS or MPEG-DASH really be the way to go or am I missing a more basic chunked file streaming mechanism with good libraries?
The browser will make ranged HTTP requests to get the data it needs out of the regular static media file. You don't need to do anything special to stream it. Just make sure your server is configured to handle ranged requests... most any should be able to do this right out of the box.
How - theoretically - would I go about limiting the amount of chunks downloaded in advance on the client side to save client-side resources, which is one of my biggest concerns?
The browser does this for you if you use an audio element. Additionally, data saving settings and the detected connectivity speed may impact whether or not the browser pre-fetches. The point is, you don't have to worry about this. You'll only be using what you need.
Just make sure you're compressing your media as efficiently as you can for the required audio quality. Use a good codec like Opus or AAC.
I was looking into hosting services as well, but figured that most of them are specialized in hosting podcasts (fewer but very large files). Has anyone an opinion about whether I could use these services to host and stream possibly 1000s of files with sizes ranging from very small to rather large?
Most any regular HTTP CDN will work just fine.
One final note for you... beware of iOS and Safari. Thanks to Apple's restrictive policies, all browsers under iOS are effectively Safari. Safari is incapable of playing more than one audio element at a time. If you use the Web Audio API you have more flexibility, but the Web Audio API has no real provision for streaming. You can use a media element source node, but this breaks lock screen metadata and outright doesn't work on some older versions of iOS. TL;DR; Safari is all but useless for audio on the web, and Apple's business practices have broken any alternatives.
Is possible to pipe raw video frames to a browser/website? For instance the decoding could be done locally in Gstreamer, and then that could be forwarded somehow to a browser.
EDIT:
I realize that my description was a bit shaky. The use case I would like to have is to send encoded video to someone, decode it on their computer, do some advance filtering that cannot be done in the browser, and then pipe the frames to the browser. Obviously re-encoding it would just be a waste of time and energy.
All that I can find is ppl saying that video frames can be grabbed FROM a browser, no-one seems to be interested to SENT TO a browser. The horrible option could be to use webrtc and to re-encode the frames into VP8 and then to send it to the browser.
So my final question is whether it is possible to write to the rendering pipeline of a browser? I know next to nothing about web programming, I usually just deal with images and video.
Thank you for your support :)
PS: forgive my lack of knowledge, is it possible to have a client on someone's computer, writting to a local tcp port, and to access that tcp port from a website in the browser? (potentially asking the user to allow the connection?)
Yes, this is possible. Since you're running a local GStreamer pipeline, you might look into this project: https://github.com/Samsung/ChromiumGStreamerBackend Basically, they're using GStreamer as the native renderer in-browser.
Aside from that, you can create a browser extension which executes an application and gets data from Gstreamer, to shuffle to your application. https://developer.chrome.com/extensions/nativeMessaging
If you don't want to make an extension, you can instead create a small Web Socket server.
Either way, you can write the raw pixel data to a Canvas... no need to re-encode/decode the video. https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API
This may be a Servlets question or an HTML5 question, depending on what the solution turns out to be... :)
I've got a (Tomcat) Servlet serving up short clips of audio which are then picked up in an HTML5 audio element. The audio correctly plays, but on some browsers only once (so that attempting to "rewind" or replay the audio does not then work). I suspect that this is because my Servlet is not reporting that it supports range requests: I notice that given a static audio file on the same server, Apache adds the HTTP "range-unit" response header, and replaying the file then works in such cases. So I am supposing that on browsers where I'm having the issue, in order to replay the file, the browser makes an HTTP Content-Range request rather than either buffering the entire file or re-requesting the entire file. (On Safari at least, replaying the audio served from my Servlet works fine: I'm guessing because Safari buffers the entire audio.)
So my questions:
is there a way in HTML to request that the browser buffer the entire audio file on playing in order to allow replays rather than the server needing necessarily to support range requests?
if not, does anyone have any experience of responding to range requests from a Servlet? I'm assuming it's just a case of (a) sending a "range-unit" response code in response to the initial request, then (b) looking out for relevant HTTP request headers ("Content-Range?") (I guess that's how they're handled?) on subsequent requests and only serving the relevant portion of the audio? Are there any pitfalls I should be aware of?
is there a way in HTML to request that the browser buffer the entire audio file on playing in order to allow replays rather than the server needing necessarily to support range requests?
Even if there is, it will not be supported on all browsers, especially mobiles/ipads/etc.
does anyone have any experience of responding to range requests from a Servlet?
There's implementation provided by BalusC. I have ported it to my environment and, apart some minor issues, (which are most likely unrelated to the implementation but to the client side specifics) it works great.
I am building an application that allows authenticated users to use a Web browser to upload MP3 audio files (of speeches) to a server, for distributing the audio on a network. The audio files need to use a specific bit rate (32kbps or less) to ensure efficient use of bandwidth, and an approved sampling rate (22.050 or 44.100) to maximize compatibility. Rather than validate these requirements following the upload using a server-side script, I was hoping to use HTML5 FileReader to determine this information prior to the upload. If the browser detects an invalid bit rate and/or sampling rate, the user can be advised of this, and the upload attempt can be blocked, until necessary revisions are made to the audio file.
Is this possible using HTML5? Please note that the question is regarding HTML5, not about my application's approach. Can HTML5 detect the sampling rate and/or bit rate of an MP3 audio file?
FYI note: I am using an FTP java applet to perform the upload. The applet is set up to automatically forward the user to a URL of my choosing following a successful upload. This puts the heavy lifting on the client, rather than on the server. It's also necessary because the final destination of each uploaded file is different; they can be on different servers and different domains, possibly supporting different scripting languages on the server. Any one server would quickly exceed its storage space otherwise, or if the server-side script did an FTP transfer, the server's performance would quickly degrade as a single point of failure. So for my application, which stores uploaded audio files on multiple servers and multiple domains, validation of the bit rate and sampling rate must take place on the client side.
You can use FileReader API and Javascript built audio codecs to extract this information from the audio files.
One library providing base code for pure JS codecs is Aurora.js - then the actual codec code is built upon it
https://github.com/audiocogs/aurora.js/wiki/Known-Uses
Naturally the browser must support FileReader API.
I didn't understand from your use case why you need Java applet or FTP. HTTP uploads work fine for multiple big files if done properly using async badckend (like Node.js, Python Twisted) and scalable storage (Amazon S3). Similar use case is resizing incoming images which is far more demanding application than extracting audio metadata out from the file. The only benefit on the client side is to reduce the number of unnecessary uploads by not-so-technically-aware users.
Given that any user can change your script/markup to bypass this or even re-purpose it, I wouldn't even consider it.
If someone can change your validation script with a bit of knowledge of HTML/Javascript, don't use HTML/Javascript. It's easier to make sure that it is validated, and validated correctly by validating it on the server.
I am currently developing a HTML5 game which loads in an external resource. Currently, I am using XMLHttpRequest to read in the file, but this does not work on Chrome, resulting in a
XMLHttpRequest cannot load file:///E:/game/data.txt
Cross origin requests are only supported for HTTP.
The file is in the same directory as the HTML5 file.
Questions:
Is there any way for a HTML5 application to use XMLHttpRequest (or
other method) to load in an external file without requiring it to be
hosted on a webserver?
If I package the HTML5 code as an application on a tablet/phone
which supports HTML5, would XMLHttpRequest be able to load external
files?
(a) Yes and no. As a matter of security-policy, XHR has traditionally been both same-protocol (ie: http://, rather than file:///), and on top of that, has traditionally been same-domain, as well (as in same subdomain -- http://pages.site.com/index can't get a file from http://scripts.site.com/). Cross-domain requests are now available, but they require a webserver, regardless, and the server hosting the file has to accept the request specifically.
(b) So in a roundabout way, the answer is yes, some implementations might (incorrectly) allow you to grab a file through XHR, even if the page is speaking in file-system terms, rather than http requests (older versions of browsers)... ...but otherwise you're going to need a webserver of one kind or another. The good news is that they're dirt-simple to install. EasyPHP would be sufficient, and it's pretty much a 3-click solution. There are countless others as well. It's just the first that comes to mind in terms of brain-off installation, if all you want is a file-server in apache, and you aren't planning on using a server-side scripting language (or if you do plan on using PHP).
XMLHttpRequest would absolutely be able to get external files...
IF they're actually external (ie: not bundled in a phone-specific cache -- use the phone's built-in file-access API for that, and write a wrapper to handle each one with the same, custom interface), AND the phone currently has reception -- be prepared to handle failure-conditions (like having a default-settings object, or having error-handling or whatever the best-case is, for whatever is missing).
Also, look into Application Cache Manifests. Again, this is an html5 solution which different versions of different phones handle differently (early-days versus more standardized formats). DO NOT USE IT DURING DEVELOPMENT, AS IT MAKES TESTING CODE/CONTENT CHANGES MISERABLY SLOW AND PAINFUL, but it's useful when your product is pretty much finished and bug-free, and seconds away from launch, where you tell users' browsers to cache all of the content for eternity, so that they can play offline, and they can save all kinds of bandwidth, not having to download everything the next time they play.