In the context of HTML5 audio, if the server is sent a request with the following header:
Range:bytes=0-
Will responding with
Content-Range: bytes 0-499/1234
Be handled correctly by all modern browsers? When byte[500] is needed, will the browser automatically know to request the next chunk? Am I interpreting the following spec correctly?
If all of the preconditions are true, the server supports the Range
header field for the target resource, and the specified range(s) are
valid and satisfiable (as defined in Section 2.1), the server SHOULD
send a 206 (Partial Content) response with a payload containing one
or more partial representations that correspond to the satisfiable
ranges requested, as defined in Section 4.
I've been reading from the following spec: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-p5-range-22
If that request was generated internally by the browser by assigning the src attribute of an audio of video tag, then yes, the media element knows how to handle byte range requests and make subsequent requests for the next segments. No additional logic needed.
It sounds like you're interpreting the spec correctly. However, examples of ServiceWorkers responding to Range Requests suggests that the expected response to open-ended requests like 0- is the entire byte range. Browsers then stream the response up to some heuristic, and if the user seeks to a range outside of what has been streamed the initiate a subsequent open-ended request.
In order words, if there's a file with 1000 bytes: the first request is always Range: bytes=0-. The browser decides to load 100 bytes. The user seeks toward the end, and the browser sends another request Range: bytes=900-.
I've actually never seen instances where browsers request explicit, closed ranges like Range: bytes=0-100. They also don't seem to use the range size, i.e. 1234 in Content-Range: bytes 0-499/1234. I'm guessing this is because some servers may send * for files of unknown duration (or continuous, real-time playback).
Word of caution: this is based on my observations, predominantly of Firefox, Chrome, and iOS Safari. Behavior may vary based on browser vendor, version, etc.
Related
I'm trying to understand the Google CDN behavior in the following scenario:
Let's assume I have a backend service serving chunked http data. For the sake of the explanation, let's assume that serving a single request takes up to 10s
Let's imagine the case where a file is requested through the CDN by a client A, and that this file is not currently cached in the CDN. The request will go to the backend service, that starts serving the file. Client A will immediately start receiving HTTP chunks
After 5s, another client B requests the same file. I can envision 3 possible behaviors, but I can't figure out how to control this through the CDN configuration:
Option a: the CDN simply pass the request to the backend service, ignoring that half of the file has already been served and could already be cached. Not desirable as the backend service will be reached twice and serve the same data twice.
Option b: the CDN puts the second request on "hold", waiting for the first one to be terminated before serving the client B from its cache (in that case, the request B does not reach the backend service). Ok, but still not amazing as client B will wait 5s before getting any http data.
Option c: the CDN immediately serves the first half of the http chunks and then serves the remaining http chunks at the same pace than request A. Ideal!
Any ideas on the current behavior ? And what could we do to get the option C, which is by far our preferred option ?
Tnx, have a great day!
Jeannot
It is important to note that GFE historically cached only complete responses and stored each response as a single unit. As a result, the current behavior will follow option A. You can take a look at this help center article for more details.
However, with the introduction of Chunk caching, which is currently in Beta, large response bodies are treated as a sequence of chunks that can each be cached independently. Response bodies less than or equal to 1 MB in size can be cached as a unit, without using chunk caching. Response bodies larger than 1 MB are never cached as a unit. Such resources will either be cached using chunk caching or not cached at all.
Only resources for which byte range serving is supported are eligible for chunk caching. GFE caches only chunk data received in response to byte range requests it initiated, and GFE initiates byte range requests only after receiving a response that indicates the origin server supports byte range serving for that resource.
To be more clear, once Chunk caching is in GA, you would be able to achieve your preferred option C.
Regarding your recent query, unfortunately, only resources for which byte range serving is supported are eligible for chunk caching at the moment. You can definitely create a feature request for your use case at Google Issue Trackers.
The good news is that chunk caching with Cloud CDN is now in GA and you can check the functionality anytime you wish.
When an image is uploaded from the client's machine to the client (browser), it requires FileReader() API in html, thereafter a base64 encoded url (say) of the image is sent in chunks to the server, where it needs to be re-assembled. All of this is taken care by the developer.
However, when an image is sent from the server to the client, only the directory path of the image inside the server machine suffices, no chunking and encoding is required.
My questions are:
1. Does the server send the image in chunks to the html file. If it does not, how does sending full images not bottle server's network? What would happen in case of large video files?
2. In what form of binary data is the image sent to the client - base64url / ArrayBuffer / binarystring / text / etc.
3. If the server does send the image in chunks, who is doing the chunking and the re-assembly on the client thereafter?
Thanks.
HTML isn't really important here. What you care about are the transport protocols used - HTTP and TCP, most likely.
HTTP isn't chunked by default, though there are advanced headers that do allow that - those are mostly used to support seeking in large files (e.g. PDF, video). Technically, this isn't really chunking - it's just the infrastructure for allowing for partial downloads (i.e. "Give me data from byte 1024 to byte 2048.").
TCP is a stream-based protocol. From programmer point of view, that's all there is to it - no chunking. Technically, though, it will process your input data and send it as distinct packets that are reassembled in-order on the other side. This is a matter of practicality - it allows you to manage data latency, streaming, packet retransmission etc. TCP handles the specifics during connection negotiation - flow control, window sizing, congestion control...
That's not the end of it, though. All the other layers add their own bits - their own ways to package the payload and split it as necessary, their own ways to handle routing and filtering, their own ways to handle congestion...
Finally, just like HTTP natively supports downloads, it supports uploading data as well. Just send an HTTP request (usually POST or PUT) and write data in a format the server understands - anything from text through base-64 to raw binary data. The limiting factor in your case isn't the server, the browser or HTTP - it's JavaScript. The basic mechanism is still the same - a request followed by a response.
Now, to address your questions:
Server doesn't send images to the HTML file. HTML only contains an URL of the image[1], and when the browser sees an URL in the img tag, it will initiate a new, separate request just for the image data. It isn't fundamentally different from downloading a file from a link. As for the transfer itself, it follows pretty much exactly the same way as the original HTML document - HTTP headers, then the payload.
Usually, raw binary data. HTTP is a text-based protocol, but it's payload can be arbitrary. There's little reason to use Base-64 to transfer image data (though in the past, there have been HTTP and FTP servers that didn't support binary at all, so you had to use something like Base-64).
The HTTP server doesn't care (with the exception of "partial downloads" mentioned above). The underlying network protocols handle this.
[1] Nowadays, there's methods to embed images directly in the HTML text, but it's of varying practicality depending on the image size, caching requirements etc.
I have uploaded an mp4 video animation to Azure Blob Storage. The headers are are all default apart from setting the Content-Type to video/mp4. The video can be accessed at http://paddingtondev.blob.core.windows.net/media/1001/animation_default_headers.mp4
I have an Azure CDN sitting over that blob storage account. The URL for the same video through the CDN is http://az593791.vo.msecnd.net/media/1001/animation_default_headers.mp4
When I access the blob-stored video through an HTML5 video element on a web page, the browser (have tested in FF and Chrome) receives the entire video in a 200 HTTP response. Further requests for that video then receive a 304 response from blob storage.
However, when you request the video through the Azure CDN, it helpfully returns it to you as a series of HTTP 206 partial responses. This is in response to the browsers specifying a Range header with the request.
However, further requests for the video through the CDN are NOT cached, and the whole video is re-downloaded by the browser (through a series of further 206 requests).
How do I ensure the video is cached? I understand the usefulness of partial responses, but in our case the video won't be seekable and we only play it when the whole file is downloaded. I can see a few approaches here, but none have helped so far:
Forbid Azure CDN from returning partial responses
Remove range header from original browser request somehow
Persuade browsers to cache 206 partial responses
I have tried adding a max-age Cache-Control header to the file but this had no impact. Ideally we wouldn't even hit Azure at all when re-loading the video (as it will never change), but I'm happy to accept the cost of the HTTP request to Azure if it subsequently returns a 304 .
Caching 206 responses is tricky. The RFC for the client requires that in order to cache the content the ETAG and the range requested must match exactly.
There are a couple of things you can check -
1) Verify that the ETAGS did not change on the request. From the description of your environment (and setting the content expiration date), this sounds unlikely, but it may be an avenue to pursue.
2) More likely is that the range requests are not lining up. A request for byte range 1000-->2000 and a second request of 1500-->2000 would not (per RFC) be served from the client cache. So you may be in a situation of seeing exactly what is supposed to happen with that particular format/client.
I'm pretty sure HTML5 only supports progressive download, so unless you wanted to reconsider the delivery this may be expected behavior.
I am downloading an image using a GET request using XmlHttpRequest with cache disabled. In IE10, when I check the network panel, I see that the Received column shows different number of bytes and the content-length header shows the correct image size. Whenever I refresh the page several times, I see that the Received column shows different byte data always.
I need how much the size of the image is and how much time it took to download it for bandwidth calculation. For the time taken, I am getting the image entry from HTML5's performance.getEntries() and checking the duration. But for the image size, should I refer to the content-length header or the bytes received shown in network panel?
The Content-Length HTTP response header tells the browser how many byes the requested resource is. In your case, it should equal the size of the image on disk.
The Received column in the IE F12 Developer Tools shows you how many bytes were transferred across the wire, which will include the HTTP header and may or may not include the actual requested resource. For example, if the resource was already located in your cache, the Content-Length HTTP response header would list the requested resource's size, while the Received column in the F12 Developer Tools would show only the number of bytes of the HTTP 304 response header.
For your bandwidth calculation, on your dev machine, you should take the Received column to calculate how many bytes per interval were sent.
You will only be able to estimate the bandwidth "in the wild" since you can't access the developer tools on random visitor's machines. In that case, you would time how long it took to download the resource via window.performance.getEntries(), aka ResourceTiming.
To determine how many bytes were transferred for that resource in the wild, you can only guess. You know it was at least as many bytes as the file size, but you can only estimate the HTTP header size since it will vary per user / browser / proxy. You will also want to ensure the resource cannot be in the user's cache, so you should use a cache-busting URL parameter and/or set the appropriate HTTP Cache-Control headers.
I'm writing a resource handling method where I control access to various files, and I'd like to be able to make use of the browser's cache. My question is two-fold:
Which are the definitive HTTP headers that I need to check in order to know for sure whether I should send a 304 response, and what am I looking for when I do check them?
Additionally, are there any headers that I need to send when I initially send the file (like 'Last-Modified') as a 200 response?
Some psuedo-code would probably be the most useful answer.
What about the cache-control header? Can the various possible values of that affect what you send to the client (namely max-age) or should only if-modified-since be obeyed?
Here's how I implemented it. The code has been working for a bit more than a year and with multiple browsers, so I think it's pretty reliable. This is based on RFC 2616 and by observing what and when the various browsers were sending.
Here's the pseudocode:
server_etag = gen_etag_for_this_file(myfile)
etag_from_browser = get_header("Etag")
if etag_from_browser does not exist:
etag_from_browser = get_header("If-None-Match")
if the browser has quoted the etag:
strip the quotes (e.g. "foo" --> foo)
set server_etag into http header
if etag_from_browser matches server_etag
send 304 return code to browser
Here's a snippet of my server logic that handles this.
/* the client should set either Etag or If-None-Match */
/* some clients quote the parm, strip quotes if so */
mketag(etag, &sb);
etagin = apr_table_get(r->headers_in, "Etag");
if (etagin == NULL)
etagin = apr_table_get(r->headers_in, "If-None-Match");
if (etag != NULL && etag[0] == '"') {
int sl;
sl = strlen(etag);
memmove(etag, etag+1, sl+1);
etag[sl-2] = 0;
logit(2,"etag=:%s:",etag);
}
...
apr_table_add(r->headers_out, "ETag", etag);
...
if (etagin != NULL && strcmp(etagin, etag) == 0) {
/* if the etag matches, we return a 304 */
rc = HTTP_NOT_MODIFIED;
}
If you want some help with etag generation post another question and I'll dig out some code that does that as well. HTH!
A 304 Not Modified response can result from a GET or HEAD request with either an If-Modified-Since ("IMS") or an If-Not-Match ("INM") header.
In order to decide what to do when you receive these headers, imagine that you are handling the GET request without these conditional headers. Determine what the values of your ETag and Last-Modified headers would be in that response and use them to make the decision. Hopefully you have built your system such that determining this is less costly than constructing the complete response.
If there is an INM and the value of that header is the same as the value you would place in the ETag, then respond with 304.
If there is an IMS and the date value in that header is later than the one you would place in the Last-Modified, then respond with 304.
Else, proceed as though the request did not contain those headers.
For a least-effort approach to part 2 of your question, figure out which of the (Expires, ETag, and Last-Modified) headers you can easily and correctly produce in your Web application.
For suggested reading material:
http://www.w3.org/Protocols/rfc2616/rfc2616.html
http://www.mnot.net/cache_docs/
You should send a 304 if the client has explicitly stated that it may already have the page in its cache. This is called a conditional GET, which should include the if-modified-since header in the request.
Basically, this request header contains a date from which the client claims to have a cached copy. You should check if content has changed after this date and send a 304 if it hasn't.
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25 for the related section in the RFC.
We are also handling cached, but secured, resources. If you send / generate an ETAg header (which RFC 2616 section 13.3 recommends you SHOULD), then the client MUST use it in a conditional request (typically in an If-None-Match - HTTP_IF_NONE_MATCH - header). If you send a Last-Modified header (again you SHOULD), then you should check the If-Modified-Since - HTTP_IF_MODIFIED_SINCE - header. If you send both, then the client SHOULD send both, but it MUST send the ETag. Also note that validtion is just defined as checking the conditional headers for strict equality against the ones you would send out. Also, only a strong validator (such as an ETag) will be used for ranged requests (where only part of a resource is requested).
In practice, since the resources we are protecting are fairly static, and a one second lag time is acceptable, we are doing the following:
Check to see if the user is authorized to access the requested resource
If they are not, Redirect them or send a 4xx response as appropriate. We will generate 404 responses to requests that look like hack attempts or blatant tries to perform a security end run.
Compare the If-Modified-Since header to the Last-Modified header we would send (see below) for strict equality
If they match, send a 304 Not Modified response and exit page processing
Create a Last-Modified header using the modification time of the requested resource
Look up the HTTP Date format in RFC 2616
Send out the header and resource content along with an appropriate Content-Type
We decided to eschew the ETag header since it is overkill for our purposes. I suppose we could also just use the date time stamp as an ETag. If we move to a true ETag system, we would probably store computed hashes for the resources and use those as ETags.
If your resources are dynamically generated, from say database content, then ETags may be better for your needs, since they are just text to be populated as you see fit.
regarding cache-control:
You shouldn't have to worry about the cache-control when serving out, other than setting it to a reasonable value. It's basically telling the browser and other downstream entities (such as a proxy) the maximum time that should elapse before timing out the cache.