I already checked my local browser settings and nothing is stopping caching (I see several other network requests in the same page load that cache successfully). Whenever I navigate away from the page and back, this resource always loads (and takes a second or two to do so). It doesn't appear to be using the cache at all. Here are my headers:
General
Request URL: https://example.website.stage/htmlpage/
Request Method: GET
Status Code: 200
Remote Address: ~omitted~
Referrer Policy: strict-origin-when-cross-origin
Response Headers
alt-svc: clear
cache-control: public, max-age=43200, immutable
content-length: 68988
content-security-policy: frame-ancestors 'self'
content-type: text/html; charset=utf-8
date: Fri, 05 Mar 2021 17:01:55 GMT
server: istio-envoy
set-cookie: session_timer=session_timer; Expires=Fri, 05-Mar-2021 17:16:55 GMT; Secure; Path=/
strict-transport-security: max-age=31536000; includeSubDomains
vary: Cookie
via: 1.1 google
x-content-type-options: nosniff
x-envoy-upstream-service-time: 2618
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
Request Headers
:authority: example.website.stage
:method: GET
:path: /htmlpage/
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cookie: ~omitted~
referer: https://example.website.stage/htmlpage
sec-ch-ua: "Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36
Figured it out. The response header vary: Cookie causes all future requests to be skip the cache if the cookie doesn't match, and since our cookie contains a timestamp, it'll never match!
Related
Im trying to implement simple scraper, however I encoutered some problem. Somehow website is refusing connections from headless chrome. This is first and the only request, there is no any javascript execution. Requests from normal chrome works well so it's definitly not a banned ip. What can be wrong here? How are they posibly detecting it?
I'm running normal headless chrome and then I replace user agent, that's all.
.\chrome.exe --headless --remote-debugging-port=9222
General:
Request URL: https://www.adidas.de/
Request Method: GET
Status Code: 403
Remote Address: 23.210.248.137:443
Referrer Policy: no-referrer-when-downgrade
Response Headers:
cache-control: max-age=0, no-cache, no-store
content-length: 1952
content-type: text/html
date: Thu, 26 Dec 2019 16:16:49 GMT
expires: Thu, 26 Dec 2019 16:16:49 GMT
pragma: no-cache
status: 403
Request Headers:
:authority: www.adidas.de
:method: GET
:path: /
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
cache-control: max-age=0
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
#RobertHarvey: Yes, you are correct: sending Accept-Language is a must for some websites. You can either do it with puppeteer via their API, or with chrome-remote-interface by intercepting requests and adding header directly.
I'm implementing a REST API that should support cross domain requests. Using CORS I want to achieve this. Almost all of my requests are 'not-simple', meaning for all non-GET requests a preflight request must be send by the browser.
To limit the amount of preflight/OPTIONS requests I try to let the browser cache the OPTIONS requests. This seems to work in Firefox and Safari, but not in Chrome. I know Chrome will only cache the preflight requests for only 10 minutes, but in my case it seems no caching takes place at all.
These are the HTTP requests and responses sent/received by Chrome:
Request:
OPTIONS /api/v1/sessions HTTP/1.1
Host: xxxxxxx
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: POST
Origin: http://localhost:8000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.107 Safari/537.36
Access-Control-Request-Headers: content-type
Accept: */*
Referer: http://localhost:8000/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Response:
HTTP/1.1 200 OK
Date: Sun, 26 Jul 2015 09:33:27 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.9
Cache-Control: private, max-age=1440, pre-check=1440
Access-Control-Allow-Origin: http://localhost:8000
Access-Control-Allow-Methods: GET,POST,PATCH,DELETE
Access-Control-Max-Age: 86400
Access-Control-Allow-Headers: content-type
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
You have Pragma: no-cache & Cache-Control: no-cache headers set in the request.
Try removing them.
Api requests by default do not set these headers, and I doubt chrome does
either.
You should check your code and find out where they are
set from.
Now, given that its working fine on other browsers, you'd better check if you have set no-cache option on Dev Tools.
Preflight caching is a known bug in 98 version.
Follow below ticket for more details
https://bugs.chromium.org/p/chromium/issues/detail?id=1298477
I have an AngularJS WebAPI application.
As far as I can understand the OPTIONS request is constructed automatically by the browser.
POST http://localhost:3048/Token HTTP/1.1
Host: localhost:3048
Connection: keep-alive
Content-Length: 78
Accept: application/json, text/plain, */*
Origin: http://localhost:2757
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://localhost:2757/Auth/login
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
grant_type=password&username=xxx%40live.com&password=xxx
Response:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 971
Content-Type: application/json;charset=UTF-8
Expires: -1
Server: Microsoft-IIS/8.0
Access-Control-Allow-Origin: *
Set-Cookie: .AspNet.Cookies=CpvxrR1gPFNs0vP8GAmcUt0EiKuEzLS1stLl-70O93wsipJkLUZuNdwC8tZc5M0o1ifoCjvnRXKjEBk3nLRbFlbldJLydW2BWonr5JmBjRjXZyKtcc29ggAVhZlc2E-3gGDlyoZLAa5Et8zrAokl8vsSoXmHnsjrxZw0VecB_Ry98Ln84UuKdeHlwSBnfaKKJfsN-u3Rsm6MoEfBO5aAFEekhVBWytrYDx5ks-iVok3TjJgaPc5ex53kp7qrtH3izbjT7HtnrsYYtcfPtmsxbCXBkX4ssCBthIl-NsN2wObyoEqHMpFEf1E9sB86PJhTCySEJoeUJ5u3juTnPlQnHsk1UTcO0tDb39g-_BD-I4FWS5GMwxLNtmut3Ynjir0GndwqsvpEsLls1Y4Pq7UuVCTn7DMO4seb64Sy8oEYkKZYk9tU4tsJuGD2CAIhdSc-lAmTAA78J5NOx23klkiuSe_SSiiZo5uRpas_1CFHjhi1c8ItEMpgeTsvgTkxafq5EOIWKPRxEHbCE8Dv106k5GlKK5BaH6z7ESg5BHPBvY8; path=/; HttpOnly
X-SourceFiles: =?UTF-8?B?QzpcR1xhYmlsaXRlc3Qtc2VydmVyXFdlYlJvbGVcVG9rZW4=?=
X-Powered-By: ASP.NET
Date: Tue, 13 Jan 2015 04:54:55 GMT
{"access_token":"TkJ2trqT ....
Now logged in
I log out which is nothing more than removing the token and log in again. Something happens that is different. Before it did not send the OPTIONS but now it does. Is there something resulting from a previous request/response that would influence the browser to act different the second time I log in?
OPTIONS http://localhost:3048/Token HTTP/1.1
Host: localhost:3048
Connection: keep-alive
Access-Control-Request-Method: POST
Origin: http://localhost:2757
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
Access-Control-Request-Headers: accept, authorization, content-type
Accept: */*
Referer: http://localhost:2757/Auth/login
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Response:
HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Expires: -1
Server: Microsoft-IIS/8.0
X-SourceFiles: =?UTF-8?B?QzpcR1xhYmlsaXRlc3Qtc2VydmVyXFdlYlJvbGVcVG9rZW4=?=
X-Powered-By: ASP.NET
Date: Tue, 13 Jan 2015 04:56:32 GMT
{"error":"unsupported_grant_type"}
If I do a browser reset and reload of the page then it goes back to like before where it does not send OPTIONS the first time and I am able to log in.
Probably I need to change something on the server so it handles options.
BUT why does my browser (Chrome) not send OPTIONS the first time?
Whether the Chrome (or any other browser) sends an OPTIONS request is exactly specified by the CORS specfication:
When the cross-origin request algorithm is invoked, these steps must be followed:
...
2. If the following conditions are true, follow the simple cross-origin request algorithm:
The request method is a simple method and the force preflight flag is unset.
Each of the author request headers is a simple header or author request headers is empty.
3. Otherwise, follow the cross-origin request with preflight algorithm.
Note: Cross-origin requests using a method that is simple with author request headers that are not simple will have a preflight request to ensure that the resource can handle those headers. (Similarly to requests using a method that is not a simple method.)
Your OPTIONS request contains the following request header:
Access-Control-Request-Headers: accept, authorization, content-type
This means that your Angular app has inserted the non-simple Authorization request header, probably as a part of an authentication scheme. Non-simple "author request headers" trigger the OPTIONS request, as you can see in the above quote.
To allow the request to succeed, your server should handle OPTIONS request and respond with:
Access-Control-Allow-Origin: https://example.com
Access-Control-Allow-Headers: authorization
To learn more about CORS, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS.
When you first login you most likely set the Authorization HTTP header somewhere in your login procedure. On the other side, you forgot to remove this header when the user logs out.
When you try to login again, the Authorization HTTP header is still present. This triggers the browser to perform a preflight request (see explanation of Rob W: https://stackoverflow.com/a/27924344/548020. Considering that you try to login with a grant type password it does not make sense to send an Authorization header, as this implies that you are already authorized (= logged in). Your are basically asking your backend to log you in and at the same time telling your backend that you are already authorized (= logged in).
This can be fixed by simple removing the Authorization HTTP header when the user logs out.
You can also clean your Headers when you login, before sending your POST request:
delete $http.defaults.headers.common['Authorization'];
I'm trying to follow Google's caching recommendation, but neither IE nor Chrome are caching my images when HTTPS is used. The second request is not even a conditional GET. If I simply switch to HTTP, it works fine.
Here's request information, according to Chrome's request logger:
Remote Address: ::1:443
Request URL: https://localhost/getmyimage.php?id=123
Request Method: GET
Status Code: 200 OK
Request Headers
Accept: image/webp,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en;q=0.8
Connection: keep-alive
Cookie: PHPSESSID=gbk4vk7ejlr20nqgajcqgskul7
Host: localhost
Referer: https://localhost/
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
Query String Parameters
id: 123
Response Headers
Cache-Control: public
Connection: Keep-Alive
Content-Length: 3224
Content-Type: image/png
Date: Tue, 27 May 2014 06:53:03 GMT
Expires: Mon, 25 Aug 2014 06:53:03 GMT
Keep-Alive: timeout=5, max=99
Last-Modified: Mon, 24 Feb 2014 02:17:21 GMT
Server: Apache/2.4.7 (Win32) OpenSSL/1.0.1e PHP/5.5.9
X-Powered-By: PHP/5.5.9
i think this is happening because of the url format, you can use apache's mod_rewrite to make a url format for images given by this script to look like localhost/image/123.png
EDIT
after reading your comment, i can say that it's not about your server's config, you can't do anything about it according to this and this because of HTTPS implementations since you have Cache-control: public set already.
Using Google Chrome I'm opening Flex4.5 Client that makes GET Http Request to Rails back end that renders back json response.
If the response is 4xx Client Error then Chrome Developer Tools network tab shows the request as canceled, and I can't access the response error message through the Fault content in Flex.
This happens only on chrome. It works fine for FF and IE or if I execute the query in the chrome browser.
Below are the request and response headers copied from Chrome Developer Tools
Thank you for your help
GET (canceled) application/json Other
Request Header
GET url HTTP/1.1
Host: localhost:3000
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.83 Safari/535.11
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Response Header
HTTP/1.1 404 Not Found
Content-Type: application/json; charset=utf-8
X-Ua-Compatible: IE=Edge
Cache-Control: no-cache
X-Runtime: 0.500000
Content-Length: 30
Server: WEBrick/1.3.1 (Ruby/1.9.2/2011-07-09)
Date: Wed, 28 Mar 2012 21:53:40 GMT
Connection: Keep-Alive