I have uploaded an mp4 video animation to Azure Blob Storage. The headers are are all default apart from setting the Content-Type to video/mp4. The video can be accessed at http://paddingtondev.blob.core.windows.net/media/1001/animation_default_headers.mp4
I have an Azure CDN sitting over that blob storage account. The URL for the same video through the CDN is http://az593791.vo.msecnd.net/media/1001/animation_default_headers.mp4
When I access the blob-stored video through an HTML5 video element on a web page, the browser (have tested in FF and Chrome) receives the entire video in a 200 HTTP response. Further requests for that video then receive a 304 response from blob storage.
However, when you request the video through the Azure CDN, it helpfully returns it to you as a series of HTTP 206 partial responses. This is in response to the browsers specifying a Range header with the request.
However, further requests for the video through the CDN are NOT cached, and the whole video is re-downloaded by the browser (through a series of further 206 requests).
How do I ensure the video is cached? I understand the usefulness of partial responses, but in our case the video won't be seekable and we only play it when the whole file is downloaded. I can see a few approaches here, but none have helped so far:
Forbid Azure CDN from returning partial responses
Remove range header from original browser request somehow
Persuade browsers to cache 206 partial responses
I have tried adding a max-age Cache-Control header to the file but this had no impact. Ideally we wouldn't even hit Azure at all when re-loading the video (as it will never change), but I'm happy to accept the cost of the HTTP request to Azure if it subsequently returns a 304 .
Caching 206 responses is tricky. The RFC for the client requires that in order to cache the content the ETAG and the range requested must match exactly.
There are a couple of things you can check -
1) Verify that the ETAGS did not change on the request. From the description of your environment (and setting the content expiration date), this sounds unlikely, but it may be an avenue to pursue.
2) More likely is that the range requests are not lining up. A request for byte range 1000-->2000 and a second request of 1500-->2000 would not (per RFC) be served from the client cache. So you may be in a situation of seeing exactly what is supposed to happen with that particular format/client.
I'm pretty sure HTML5 only supports progressive download, so unless you wanted to reconsider the delivery this may be expected behavior.
Related
I have a SPA which consumes some static assets from the backend server. For reasons, I picked ETag validation as the caching mechanism. In short, I want the browser keep the assets in its cache forever, as long as the related ETags remain unchanged.
To signal the browser about caching, header Cache-Control must be present in the the responses. To me it's absolutely comprehensible, but what makes me confused is that I have to provide max-age in the header as well. In other words Cache-Control=public doesn't work whereas Cache-Control=public, max-age=100 is the correct header.
To me it sounds contradictory. The browser inquiries the server to see if an asset has changed using If-Not-Match={ETag} any time it asks for it. What's the role of max-age here then?
The resource/file cached in browser with ETag will be anyway requested each time. If this is a *.js file that was changed on server then server will send a new version with a new ETag and browser will refresh it's cached version.
But anyway performed a full network round trip of request and response and this is quite expensive.
If you do expect that some file really may change at any time then you have to use ETag.
The Cache-Control is a directive to a browser to not even try to retrieve an updated version for some time specified by the max-age. This is much more performant.
This is useful for static assets that probably wont be changed e.g. jquery-3.1.js
file will be always the same.
Or even if the resource was changed it's not a big deal e.g. style.css.
During development when assets often changed the Cache-Control is usually disabled.
But please be careful with the public modifier: that means that the resource may be cached on a proxy server (like CloudFlare) and shared between different users. If the resource have private info e.g. messages then users may see data of each others.
I'm trying to understand the Google CDN behavior in the following scenario:
Let's assume I have a backend service serving chunked http data. For the sake of the explanation, let's assume that serving a single request takes up to 10s
Let's imagine the case where a file is requested through the CDN by a client A, and that this file is not currently cached in the CDN. The request will go to the backend service, that starts serving the file. Client A will immediately start receiving HTTP chunks
After 5s, another client B requests the same file. I can envision 3 possible behaviors, but I can't figure out how to control this through the CDN configuration:
Option a: the CDN simply pass the request to the backend service, ignoring that half of the file has already been served and could already be cached. Not desirable as the backend service will be reached twice and serve the same data twice.
Option b: the CDN puts the second request on "hold", waiting for the first one to be terminated before serving the client B from its cache (in that case, the request B does not reach the backend service). Ok, but still not amazing as client B will wait 5s before getting any http data.
Option c: the CDN immediately serves the first half of the http chunks and then serves the remaining http chunks at the same pace than request A. Ideal!
Any ideas on the current behavior ? And what could we do to get the option C, which is by far our preferred option ?
Tnx, have a great day!
Jeannot
It is important to note that GFE historically cached only complete responses and stored each response as a single unit. As a result, the current behavior will follow option A. You can take a look at this help center article for more details.
However, with the introduction of Chunk caching, which is currently in Beta, large response bodies are treated as a sequence of chunks that can each be cached independently. Response bodies less than or equal to 1 MB in size can be cached as a unit, without using chunk caching. Response bodies larger than 1 MB are never cached as a unit. Such resources will either be cached using chunk caching or not cached at all.
Only resources for which byte range serving is supported are eligible for chunk caching. GFE caches only chunk data received in response to byte range requests it initiated, and GFE initiates byte range requests only after receiving a response that indicates the origin server supports byte range serving for that resource.
To be more clear, once Chunk caching is in GA, you would be able to achieve your preferred option C.
Regarding your recent query, unfortunately, only resources for which byte range serving is supported are eligible for chunk caching at the moment. You can definitely create a feature request for your use case at Google Issue Trackers.
The good news is that chunk caching with Cloud CDN is now in GA and you can check the functionality anytime you wish.
According to the HTTP spec, upon loading a resource that results in a 302 redirect:
...the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests. This response is only cacheable if indicated by a Cache-Control or Expires header field.
However, within a single page load, I'm seeing current Chrome and Firefox both resolving subsequent requests to the initial Request-URI to the resolved value from the first request, even when the redirect specifies no caching.
I've setup a minimal repro case here:
http://chrome-302-broke.herokuapp.com/test.html
It's on a free heroku dyno (in case you reach it while it's offline).
Am I missing something? It seems like caching the redirect from the initial response, even within the same page load, is taking liberty with the description from the spec. A strict interpretation shouldn't cache this request at all.
Especially with a growing number of web applications that don't navigate between pages for a considerable amount of time, this seems like it would cause problems for an increasing number of use cases.
Is this something I should submit as a bug to Chrome/Firefox?
I'm sending from my server partial contents (206) with length 1mb. But video player does not make buffer this and make new request only if SHOWED this partial content. And if client has slow internet, he get delay, while new request executed. How may make buffer partial content? I would that browser make new request, when partial content full LOADED, but not SHOWED else.
A 206 can only be returned when the browser asks for a range via the Content-Range header. Then the server must fulfill the requested range exactly as requested with a 206. If the browser does not specify a Content-Range header, returning a 206 is illegal. You, as the server can not force the browser to make a range request, but you can tell it your server can accept ranges via the Accept-Ranges header.
I'm using Node.js, Express (and connect), and fluent-ffmpeg.
We want to stream audio files that are stored on Amazon S3 through http.
We have all working, except that we would like to add a feature, the on-the-fly conversion of the stream through ffmpeg.
This is working well, the problem is that some browsers checks in advance before actually getting the file.
Incoming requests containing the Range header, for which we reply with a 206 with all the info from S3, have a fundamental problem: we need to know in advance the content-length of the file.
We don't know that since it is going through ffmpeg.
One solution might be to write out the resulting content-length directly on S3 when storing the file (in a special header), but this means we have to go through the pain of having queues to encode after upload just to know the size for future requests.
It also means that if we change compressor or preset we have to go through all this over again, so it is not a viable solution.
We also noticed big differencies in the way Chrome and Safari request the audio tag src, but this may be discussion for another topic.
Fact is that without a proper content-length header in response everything seems to break or browsers goes in an infinite loop or restart the stream at pleasure.
Ideas?
This seems to be working for me.
It would be great if you could confirm whether it gives the expect results in your browsers too.
res.writeHead(200, {
'Transfer-Encoding': 'chunked'
, 'Content-Type': 'audio/mpeg'
, 'Accept-Ranges': 'bytes' //just to please some players, we do not actually allow seeking
});
Basically, you tell the browser that you are going to stream using chunked encoding. An issue may be that some browsers do not like streaming without know how much bytes they should expect in total.