I am getting (Blocked Web browser is not ready to send) on pingdom on all my scrips and images in the page. What is coursing this? and is it something I can do about it?
As Nathan said, browsers have a hard limit for concurrent connections per hostname; Chrome I think is about six. You have three approaches: to work around that limit, to increase it, or to make it irrelevant.
You can combine all your assets so the need for request matches the browser limit (minify css, combine JS etc.)
You can "increase" the limit by using domain sharding, that means to load your assets from different subdomains for example.
images.yourdomain.com
css.yourdomain.com
The problem is that this increases DNS resolution times, so it's convenient only in certain situations.
You can enable protocols like http2 which opens one connection per domain, but internally uses a multiplexing feature allowing it to make a much larger number of requests, depending on how that limit was negotiated between the client and the server.
Here's the answer I got from Pingdom support:
"It means that the web browser is waiting for other requests to complete before issuing a new one. A web browser is configured to have a maximum number of simultaneous connections to a single domain, so if there are many such connections/requests at the same time it can cause a block and slowdown of the page."
And one way to fix it would be, "Decreasing the number of components on a page reduces the number of HTTP requests required to render the page, resulting in faster page loads. Some ways to reduce the number of components include: combine files, combine multiple scripts into one script, combine multiple CSS files into one style sheet, and use CSS Sprites and image maps." (from the Pingdom tip "Make fewer HTTP requests").
Related
I have a website that is used as a kiosk app. When online, I preload data and images from a wordpress API, and store the images in the cache storage.
A service worker intercepts http gets to these images and serves the cache data instead. Like this, the app can run offline (API calls included).
But after few hours running offline (generally around 6h) some images disappear from the cache storage. And it's always the same ones.
But not all.
Any idea where should I check to see what's going wrong ?
There can be two possible reasons for such clearing of cache, either your storage gets full(chrome localStorage), so it gets clear OR check for the expiring length of the data send from the server, check for headers that might give you insights of time it takes to expire.
And for checking if the data gets evicted from the storage, try testing your website in Safari or Edge browser where such eviction does not occur.
How did you configured Cache-Control? The ones which are constantly deleted might configured as max-age.
It's still recommended that you configure the Cache-Control headers on
your web server, even when you know that you're using the Cache
Storage API. You can usually get away with setting Cache-Control:
no-cache for unversioned URLs, and/or Cache-Control: max-age=31536000
for URLs that contain versioning information, like hashes.
as stated at https://web.dev/service-workers-cache-storage
Also, you should have a check storage amount of cache storage. (https://developers.google.com/web/tools/chrome-devtools/storage/cache) Even they claim that either all caches are deleted or none of them, since currently it is experimental technology, it is normal to be suspicious about this point.
You are also responsible for periodically purging cache entries. Each
browser has a hard limit on the amount of cache storage that a given
origin can use. Cache quota usage estimates are available via the
StorageEstimate API. The browser does its best to manage disk space,
but it may delete the Cache storage for an origin. The browser will
generally delete all of the data for an origin or none of the data for
an origin. Make sure to version caches by name and use the caches only
from the version of the script that they can safely operate on. See
Deleting old caches for more information.
as stated at https://developer.mozilla.org/en-US/docs/Web/API/Cache
Service Workers use the Storage API to cache assets.
There are fixed quotas available and different from browser to browser.
You can get some more hints from your app using the following call and analysing the results:
navigator.storageQuota.queryInfo("temporary")
.then(function(info) {
console.log(info.quota);
// It gives us the quota in bytes
console.log(info.usage);
// It gives us the used data in bytes
});
UPDATE 1
Do you have anywhere a cleanup method or logic that removes old cache entries and eventually is triggered somehow unexpected? Something like the code below.
caches.keys().then(function(keyList) {
return Promise.all(keyList.map(function(key) {
if (key !== cacheName) {
return caches.delete(key);
}
}));
})
A couple of more questions:
Do you have any business logic in the SW dealing with caching the images?
Could you get any info by using the storageQuota methods to see the available quota and usage? This might be very useful to understand if the images are evicted because of reaching the limit. (even by setting a box as persistence, you are not 100% sure the assets are retained)
From "Building PWAs" (o'Reilly - Tal Alter):
Each browser behaves differently when it comes to managing CacheStorage, allocating space in the cache for each site, and clearing out old cache entries.
In addition to a storage limit per site (also known as an origin), most browsers will also set a size limit for their entire cache. When the cache passes this limit, the browser will delete the caches of the site accessed the longest time ago (also known as the least recently used).
Browsers will never delete only part of your site’s cache. Either your site’s entire cache will be deleted or none of it will be. This ensures your site never has an unpredictable partial cache.
Considering the last words "Browsers will never delete only part of your site’s cache." might me think that maybe it is not a problem with the cache limit reached, as otherwise all images wiould be deleted in the cache.
If interested, I wrote an article about this topic here.
You could mark your data storage unit (aka "box") as "persistent", making the user agent retain it as long as possible (more details in the Storage API link above).
I am developing a Chrome App that needs to work off-line, i.e., I want it to use the App Cache, rather than sending a request to the server, as much as possible. However, should it be necessary to hit the server, then I would like to include a few bytes of data (so the server can collect and analyse some statistics about, for example, how many previous requests were served by the cache alone).
I can not send the data bytes in the URL query, because this defeats the cache (and is a well known technique for ensuring the cache is bypassed - exactly what I do NOT want to do).
I can't use a POST request, as that will also defeat the cache.
I have thought about including data in a header, but am unsure how, and unsure if this will do the trick as everything I have found about this idea recommends against it. I do not want to get in the trap of relying on something completely undocumented in case it stops working in the future (and I am not sure what will work today).
Can I include data in a GET request that will have absolutely no effect on the App Cache (in Chrome in particular), but is available to the server if and when the request makes it that far?
See comment above; this question isn't really about Chrome Apps. It's more about webapps that use App Cache.
It might be helpful to decouple normal web requests from the reporting feature. You are trying to piggyback the stats on the normal requests, but there doesn't seem to be a need to do so. Moreover, for the occasional client that caches perfectly, you'll never hear from that client because the reporting never occurs. So even in the best case the design would have been flaky.
Instead, you could increment a counter every time you make a request that might be cacheable, and keep it somewhere persistent like in localStorage. Then periodically report it to a noncached destination. You can then subtract the traffic you are receiving at the web server (which represents uncached requests) from that reported total (which represents all requests, cached or not). The difference is cached requests.
As far as I know, at the current moment, late 2011 the max-connections-per-server limit remains 6. Please correct me if I am wrong. This is bad that we cannot fix this easily as in Firefox. As far as I know this value is hardcoded.
One of the solutions is to download the Chromium's sources and rebuild them. Is there a more easy solution?
Is there any tricky way to hack this without creating a dozen of mirror-domains?
Why I'm asking the question: My task is to create a html-javascript slideshow that will run inside a fullscreened browser, and a huge monitor is hanging on the wall. The javascript is really complicated, it preloads photos and makes a lot of ajax calls to my web services. If WIFI connection is slow, if 6 photos are loading, the AJAX calls fail, the application runs bad. I want a fast solution based, on http or browser or ubuntu tweak something else, because rebuilding the javascript app will take days.
Offtopic: do you know any other things that can be tweaked in my concrete situation?
IE is even worse with 2 connection per domain limit. But I wouldn't rely on fixing client browsers. Even if you have control over them, browsers like chrome will auto update and a future release might behave differently than you expect. I'd focus on solving the problem within your system design.
Your choices are to:
Load the images in sequence so that only 1 or 2 XHR calls are active at a time (use the success event from the previous image to check if there are more images to download and start the next request).
Use sub-domains like serverA.myphotoserver.com and serverB.myphotoserver.com. Each sub domain will have its own pool for connection limits. This means you could have 2 requests going to 5 different sub-domains if you wanted to. The downfall is that the photos will be cached according to these sub-domains. BTW, these don't need to be "mirror" domains, you can just make additional DNS pointers to the exact same website/server. This means you don't have the headache of administrating many servers, just one server with many DNS records.
I don't know that you can do it in Chrome outside of Windows -- some Googling shows that Chrome (and therefore possibly Chromium) might respond well to a certain registry hack.
However, if you're just looking for a simple solution without modifying your code base, have you considered Firefox? In the about:config you can search for "network.http.max" and there are a few values in there that are definitely worth looking at.
Also, for a device that will not be moving (i.e. it is mounted in a fixed location) you should consider not using Wi-Fi (even a Home-Plug would be a step up as far as latency / stability / dropped connections go).
BTW, HTTP 1/1 specification (RFC2616) suggests no more than 2 connections per server.
Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.
There doesn't appear to be an external way to hack the behaviour of the executables.
You could modify the Chrome(ium) executables as this information is obviously compiled in. That approach brings a lot of problems with support and automatic upgrades so you probably want to avoid doing that. You also need to understand how to make the changes to the binaries which is not something most people can pick up in a few days.
If you compile your own browser you are creating a support issue for yourself as you are stuck with a specific revision. If you want to get new features and bug fixes you will have to recompile. All of this involves tracking Chrome development for bugs and build breakages - not something that a web developer should have to do.
I'd follow #BenSwayne's advice for now, but it might be worth thinking about doing some of the work outside of the client (the web browser) and putting it in a background process running on the same or different machines. This process can handle many more connections and you are just responsible for getting the data back from it. Since it is local(ish) you'll get results back quickly even with minimal connections.
I maintain a local intranet site that among other things, displays movie poster images from IMDB.com. Until recently, I simply had a perl script download the images I needed and save them to the local server. But that became a HUGE space-hog, so I thought I could simply point my site directly to IMDB servers, since my traffic is very minimal.
The result was that some images would display, while others wouldn't. And images that were displayed, would sometimes disappear after a few refreshes. The images existed on the IMDB servers, they just wouldn't display on my page.
It seems unlikely to me that IMDB would somehow block this kind of access, but is that possible? Is there something that needs to be configured on my end?
I'm out of ideas - it just doesn't make sense to me.
I'm serving my pages with mod_perl and HTML::Mason, if that's relevant.
Thanks,
Ryan
Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_perl/2.0.4 Perl/v5.10.0
Absolutely they would block that kind of access. You're using their bandwidth, which they have to pay for, for your web site. Sites will often look at the referrer, see that its not coming from their site, and either block or throttle access. Likely you're seeing this as an intermittent problem because IMDB is allowing you some amount of use of their images.
To find out more, look at the HTTP logs on your client. Either by using a browser plugin or by scripting it. Look at the HTTP response codes and you'll probably see some 4xx or 5xx responses.
I would suggest either caching the images in a cache that expires unused images, that will balance accesses with space, or perhaps getting a paid IMDB account. You may be able to get an API key to use to fetch images indicating you are a paying customer.
IMDB sure could be preventing your 'bandwidth theft' by checking the "referer". More info here: http://www.thesitewizard.com/archive/bandwidththeft.shtml
Why is it intermittent? Maybe they only implement this on some of the servers in their web farm.
Just to add to the existing answers, what you're doing is called "hotlinking", and people who run websites don't like it very much. Google for "hotlink blocking".
Well, it's more to do with images and memory in general. If I use the same image multiple times on a page, will each image be consolidated in memory? Or will each image use a separate amount of memory?
I'm concerned about this because I'm building a skinning system for a Windows Desktop Gadget, and I'm looking at spriting the images in the default skin so that I can keep the file system looking clean. At the same time I want to try and keep the memory footprint to a minimum. If I end up with a single file containing 100 images and re-use that image 100 times across the gadget I don't want to have performance issues.
Cheers.
What about testing it? Create a simple application with and without spriting, and monitor your windows memory to see which approach is better.
I'm telling you to test it because of this interesting post from Vladimir, even endorsed by Mozilla "use sprites wisely" entry:
(...) where this image is used as a sprite. Note that this is a 1299x15,000 PNG.
It compresses quite well — the actual download size is around 26K - but browsers
don't render compressed image data. When this image is downloaded and
decompressed, it will use almost 75MB in memory (1299 * 15000 * 4).
(At the end of Vladimir's post there are some other great references to check)
Since I don't know how Windows renders it's gadgets (and if it's not going to handle compressed image data), it's dificult IMHO to say exactly which approach is better without testing.
EDIT: The official Windows Desktop blog (not updated since 2007) says the HTML runtime used for Windows Gadgets is MSHTML, so I think a test is really needed to know how your application would handle the CSS sprites.
However, if you read some of the official Windows Desktop Gadgets and Windows sidebar documentation, there's an interesting thing about your decision to not use css sprites, in the The GIMAGE Protocol section:
This protocol is useful for adding
images to the gadget DOM more
efficiently than the standard HTML
tag. This efficiency results
from improved thumbnail handling and
image caching (it will attempt to use
thumbnails from the Windows cache if
the requested size is smaller than 256
pixels by 256 pixels) when compared
with requesting an image using the
file:// or http:// protocols. An added
benefit of the gimage protocol is that
any file other than a standard image
file can be specified as a source, and
the icon associated with that file's
type is displayed.
I would try to use this protocol instead of CSS sprites and do some testing too.
If none of this information would help you, I would try to ask at Windows Desktop Gadgets official forums.
Good luck!
The image will show up one time in the cache (as long as the url is the same and there's no query string appended to the file name). Spriting is the way to go.
Webbrowsers identifies cacheable resources by their ETag response header. If it is absent or differs among requests, then the image may be downloaded and stored in cache multiple times. If you (actually, the webserver) supply an unique and the same ETag header for each unique resource, then any decent webbrowser is smart enough to keep one in cache and reuse it as long as its Expires header allows.
Any decent webserver will supply the ETag header automatically for static resources, it is often autogenerated based on a combination of the local filename, the file length and the last modified timestamp. But often they don't add the Expires header, so you need to add it yourself. After judging your post history here at Stackoverflow I safely assume that you're familiar with Apache HTTPD as web server, so I'd suggest to have a look at mod_expires documentation to learn how to configure it yourself to an optimum.
In a nutshell, serve the sprite image along with an ETag and a far future Expires header and it'll be okay.