We are having an ongoing issue with a web application that we have developed where when we deploy a new version of the application, Chrome seems to ignore all the Cache-Control headers and load the application (index.html) out of disk cache, causing all kinds of errors for our users.
I've tried several different variations of the Cache-Control header, but it never seems to obey, including Cache-Control: max-age: 0, must-revalidate and Cache-Control: no-cache.
My questions are two-fold: Is there something that I'm missing that might be causing this, and are there techniques that others are using to avoid this sort of problem.
The correct max-age syntax should be
Cache-Control: max-age=0, must-revalidate
You can also try
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
which basically cover all don't-cache directives, in case some proxy in between does not support newer syntax.
Related
I'm trying to scrape products from a site (e.g. https://www.violetgrey.com/en-us/shopping/the-rich-cream-18105401). Whilst on browser it loads normally, when I copy the initial curl request for the site, it gives me access denied. This is all done in local environment. So far, before copying the curl request from browser dev tools I have:
Disabled JS for the site
Cleared all my cache, cookies
Tried different browsers
Still, it's the same result - blocked via curl. When the exact same request worked in my browser. Could anyone please point me to the right direction?
If you look at the response header you can see it comes from Cloudflare.
Cloudflare is evil. IMHO.
The HTTP status is 403. HTTP/2 403 Which means Forbidden.
The body is the text:
error code: 1020
Error 1020 can be roughly translated to "take your curl and go elsewhere. You and your curl are not wanted here."
Cloudflare profiles and fingerprints Browsers. For example they monitor the SSL/TLS handshaking and if your curl handshaking is not do the handshaking exactly like the Browser in your User Agent, they give you a 403 Forbidden and Error code 1020.
And your request does not reach violetgrey.com. They do not even know you tried.
Cloudflare is political and blocks whatever traffic they want to. If it is in their best interest not allow you through, they block you. For example Cloudflare blocked me from accessing the US Patent and Trademark site. Not only that but they sent out 3 XHR beacon requests to YouTube and Google Play. My Firefox blocked those requests. Cloudflare and Google are closely related. I do not trust either one of them.
There is no shortage of articles about your problem and possible fixes. Just search "Cloudflare 403 forbidden 1020 error". And maybe not use Google to do the search.
Here is my effort to scrape your URL. I tried a few things like trying various User Agents. I tried wget.
Request header
GET /en-us/shopping/the-rich-cream-18105401 HTTP/2
Host: www.violetgrey.com
mozilla/5.0 (x11; netbsd amd64; rv:16.0) Gecko/20121102 Firefox/16.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
alt-used: www.violetgrey.com
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: cross-site
sec-fetch-user: ?1
te: trailers
Response header:
HTTP/2 403
date: Thu, 27 Oct 2022 23:56:19 GMT
content-type: text/plain; charset=UTF-8
content-length: 16
x-frame-options: SAMEORIGIN
referrer-policy: same-origin
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
server-timing: cf-q-config;dur=4.9999998736894e-06
vary: Accept-Encoding
server: cloudflare
cf-ray: 760f5e1ced6e8dcc-MIA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
Response Body:
error code: 1020
If you are using cache.addAll or cache.add in your Service Worker, pay attention to your Cache-Control configuration in your .htaccess or elsewhere.
With the new version of Chrome 85, using the no-store value for Cache-Control prevents Service Worker from cache files.
Furthermore, the Service Worker fails to install and your website/application could be compromised.
Edit: This is likely due to a Chrome bug that has since been resolved.
That's not the expected behavior. I don't see any mentioned of "no-store" in the Service Worker Specification, and I don't see anything in ChromeStatus indicating that that change was made.
I also can't reproduce the issue on Chrome 85.0.4183.83 for the Mac.
Running the following code in the JS console, for instance, leads to a successful cache addition:
cache = await caches.open('test-cache');
await cache.addAll([
'https://httpbin.org/response-headers?Cache-Control=no-cache, no-store, must-revalidate&X-Test=1',
'https://httpbin.org/response-headers?Cache-Control=no-cache, no-store, must-revalidate&X-Test=2',
'https://httpbin.org/response-headers?Cache-Control=no-cache, no-store, must-revalidate&X-Test=3',
'https://httpbin.org/response-headers?Cache-Control=no-cache, no-store, must-revalidate&X-Test=4',
'https://httpbin.org/response-headers?Cache-Control=no-cache, no-store, must-revalidate&X-Test=5',
]);
What makes you think it's not working, and can you point to a reproduction of the issue?
I am reasonably new to browser caching. I am attempting to get Chrome to permanently cache any static file with a query parameter (for cache busting purposes). I have set Cache-Control and Expires headers way into the future, which should be adequate to say "cache this forever". The resulting response headers:
HTTP/1.1 200 OK
Cache-Control: public, max-age=315360000
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/x-javascript
Date: Wed, 16 Jul 2014 09:29:54 GMT
Last-Modified: Wed, 16 Jul 2014 03:44:14 GMT
Server: nginx/1.6.0
Transfer-Encoding: chunked
Vary: Accept-Encoding
Firefox and Safari seem to respect this for all cachebusted (?v= query parameter) files. Chrome mostly follows the directives, except for Javascript. Most of the time it does a request with an If-Modified-Since header rather than loading from cache. Sometimes one of them will load from cache and the other will yield a request resulting in a 304. Usually when loading the page from a new tab it will load from cache, but not if you hit enter in the address bar.
I've observed other websites using what I think are the exact same headers, and the files are always loaded from cache. Some of them load from cache even if you do a refresh.
I understand cache behaviour is somewhat unpredictable, but I want to make sure I'm not overseeing something that's making Chrome do that?
I had the same issue with chrome and after some hours of trial and error I figuered out, that chrome seems to have a problem with the Vary Header
I've got this snippet in my Apache / .htaccess config and as soon as I comment the line "Header append Vary Accept-Encoding" Chrome starts caching .js and .css files
<FilesMatch "(\.js\.gz|\.css\.gz)$">
# Serve correct encoding type.
Header set Content-Encoding gzip
# Force proxies to cache gzipped & non-gzipped css/js files separately.
#Header append Vary Accept-Encoding
</FilesMatch>
It still does not work while running the request via our nignx server, because it is adding the Vary: Accept-Encoding header too, when delivering gzip compressed.
So far I can guess this is a problem that only happens with chrome and as a workaround I would change the configuration to append the header only if chrome (haven't checked for safari) is not the client until there is a better fix:
<FilesMatch "(\.js\.gz|\.css\.gz)$">
# Serve correct encoding type.
Header set Content-Encoding gzip
# Force proxies to cache gzipped & non-gzipped css/js files separately.
BrowserMatch "Chrome" ChromeFound
Header append Vary Accept-Encoding env=!ChromeFound
</FilesMatch>
We have an ASP.net website, (https://website.example.com) which is loading external libraries (css and javascript) from a different sub-domain name (https://library.example.com)
The resources we are loading via the library project are only css files and javascript plugins, which themselves doesn't make any request (via AJAX).
Testing the website in normal environments, everything works fine.
However, opening it from an Internet Explorer 8 browser, returns an error:
internet explorer has modified this page to prevent cross site scripting
Could the fact that we are referencing external resources cause the error?
If yes, what would be the solution to fix this problem?
I think 90% of the websites downloads references from external domains (like CDN servers) for example.
Here's one way- configure the X-XSS-Protection header on your server.
This will tell IE to disable XSS protection on your site.
Looks something like this :
GET / HTTP/1.1
HTTP/1.1 200 OK
Date: Wed, 01 Feb 2012 03:42:24 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=6ddbc0a0342e7e63:FF=0:TM=1328067744:LM=1328067744:S=4d4farvCGl5Ww0C3; expires=Fri, 31-Jan-2014 03:42:24 GMT; path=/; domain=.google.com
Set-Cookie: NID=56=PgRwCKa8EltKnHS5clbFuhwyWsd3cPXiV1-iXzgyKsiy5RKXEKbg89gWWpjzYZjLPWTKrCWhOUhdInOlYU56LOb2W7XpC7uBnKAjMbxQSBw1UIprzw2BFK5dnaY7PRji; expires=Thu, 02-Aug-2012 03:42:24 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
1000
Please read here for more details
Still facing a weird issue with Google Chrome. I have a text/html page generated from a php source code. This page loads well and is well displayed on any popular browsers but Chrome. When the source code has just been saved, Chrome loads and render the file the right way. Even if I simply added or removed a space character. Next, if I try to refresh the page, Chrome displays a blank page and shows an error in the Developper Tools panel (see screenshot) indicating a "failed" status. But if I check the HTTP response headers, everything seems to be fine, including the HTTP status: 200 OK.
HTTP/1.1 200 OK
Date: Mon, 17 Sep 2012 08:37:03 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.3.14
Expires: Mon, 17 Sep 2012 08:52:03 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Bellow is the HTTP response headers just after saving the source and getting a correct rendering. No change (except time related information)
HTTP/1.1 200 OK
Date: Mon, 17 Sep 2012 08:56:06 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.3.14
Expires: Mon, 17 Sep 2012 09:11:07 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
I also checked the HTTP request headers, they are the same in both cases:
Working case:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Cookie:PHPSESSID=qn01olb0lkgh3qh7tpmjbdbct1
Host:(hidden here, but correct, looks like subsubdomain.subdomain.domain.tld)
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1
Failing case:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Cookie:PHPSESSID=qn01olb0lkgh3qh7tpmjbdbct1
Host:(also hidden)
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1
I noticed that even if the page failed, some other resources (javascript, style sheets) are successfully loaded or retrieved from the local cache. I can also access the HTML source code successfully every time, could the page be rendered or not (the HTML code is what it's expected to contain).
I also ran Wireshark in order to see if there would be something wrong while transferring data or something, but everything seems to be OK on this side too.
I read something on Google about content-length that would make Chrome fail if the information provided in the HTTP headers differs from the effective size of the delivered file. It does not seem to me it would be the case here as content-length is not provided.
Any help would be welcome! Thanks!
To those who are still encountering the issue:
add
setEnv downgrade-1.0
to your .htaccess, for me it works as temporary fix. It disables chunked transfers.
Another solution is to set user agent to IE, apparently it does the same...
Actual issue is somehow related to server configuration, I am also seeing segmentation fault errors.
Site I am encountering this behaviour runs on PHP 5.3.27 over HTTPS.
edit: It doesn't matter if site is accessed over HTTPS or HTTP.
edit2: The cause for this error was one extremely long line, splitting it into multiple lines solved the problem.