I store urls in a database for the users of my webapp and I am not sure whether I need to store whether it was a "http" or a "https" request.
If I don't store the type of the connection and just echo to the users a link with "http", will it in 100% of the cases use a "https" connection automatically (when it is possible)? I don't want to be responsible for a user not using a https connection even though it is possible.
Does the browser (Chrome/Firefox) automatically use https even when you try to use http?
No. If you tell the browser to use HTTP, then it will use HTTP.
Schemes will only be added to a URL automatically under two circumstances:
When it is a relative URL, in which case the scheme will be the same as the one used to load the current document.
When the user types the URL into the browser's address bar and omits the scheme, in which case it will default to HTTP (not HTTPS).
A web server might provide HTTP and HTTPS versions of the same URL with the HTTP version containing a redirect to the HTTPS version and the HTTPS version hosting the content.
A web server might, for that matter, not provide an HTTP version at all… but that is very uncommon.
I am not sure whether I need to store whether it was a "http" or a "https" request.
You should store the full URL. You shouldn't omit bits and hope that you can fill them in by guesswork.
It won't automatically do that, but there are ways to help out:
some users may have the "HTTPSeverywhere" extension, which will attempt to redirect to HTTPS
you can serve HSTS headers, which will make the browser automatically stick to HTTPS if the user has at least once been on HTTPS with your site
Now there are a few problems with these points:
not everyone use the extension
HSTS only works once the user was visiting the URL with HTTPS and it will only work on site with HSTS headers set up, so if links are external, this might not be the case.
That being said: Are the links you store links to your own domain or external links to any web site?
Related
I have a system whenever user upload an image, it will send an email to the registered user's gmail. But in the email, i see something like this, the thumbnail is not viewable.
I inspect on the element, and found the src linked to this url:
https://ci5.googleusercontent.com/proxy/VI2cPXWhfKZEIarh-iyKNz1j9q7Ymh8ty4Yz19lXh82RjSlACBzS0aRajfIj913uXAsX2ylcLEDs5FBsj4cR9TcU75Pw5djdHx4htxdCAQxs_ue1Q1wi5TV43uLLBpigpjH1xN747mUHSRdTBJmXQWFyykInJCRXicM1KhNk=s0-d-e1-ft#https://www.somedomain.com/files/1658/thumbnail_71JtDozxS1L._SY450_.jpg
Obviously it is being cached by google proxy
But i can view the image without google user content, by accessing https://www.somedomain.com/files/1658/thumbnail_71JtDozxS1L._SY450_.jpg (i masked the domain so the image might not available to you).
I tried to clear browser cache but the problem still persist. How can i bypass the googleusercontent thingy or at least make the thumbnail able to display.
I checkout on this link Images not displayed for Gmail but im not using localhost and the image itself is accessible outside of my local network.
How does Google Image Proxy work
The Google Image Proxy is a caching proxy server. Every time an image link is included in email the request will go to the Google Image Proxy first to see if it has been cached, if so it should serve it up from the proxy or it will go fetch it and cache it there after.
The solution for most issues
The Google Image Proxy server will fetch your images if this images:
have extensions like .png, .jpg/.jpeg or .gif only. May be .webp too. But not .svg.
do not use any kind of query string part in the image URL like ?id=123
have an URL which is mapped onto the image directly.
have not a long name.
Requirements for image server:
The response from image server/proxy server must include the correct header like Content-Type: image/jpeg.
File extension and content-type header must be in the same type.
Status code in server response must be 200 instead of 403, 500 and etc.
What could help too?
Google support answer:
Set up an image URL proxy whitelist
When your users open email messages, Gmail uses Google’s secure proxy
servers to serve images that might be included in these messages. This
protects your users and domain against image-based security
vulnerabilities.
Because of the image proxy, links to images that are dependent on
internal IPs and sometimes cookies are broken. The Image URL proxy
whitelist setting lets you avoid broken links to images by creating
and maintaining a whitelist of internal URLs that'll bypass proxy
protection.
When you configure the Image URL proxy whitelist, you can specify a
set of domains and a path prefix that can be used to specify large
groups of URLs. See the guidelines below for examples.
Configure the Image URL proxy whitelist setting:
Sign in to your Google Admin console. Sign in using your administrator account (does not end in #gmail.com).
From the Admin console Home page, go to Apps > G Suite > Gmail > Advanced settings. Tip: To see Advanced settings,
scroll to the bottom of the Gmail page.
On the left, select your top-level organization.
Scroll to the Image URL proxy whitelist section.
Enter image URL proxy whitelist patterns. Matching URLs will bypass image proxy protection. See the guidelines below for more details and
instructions.
At the bottom, click Save.
It can take up to an hour for changes to propagate to user accounts.
You can track prior changes under Admin console audit log.
Guidelines for applying the Image URL proxy whitelist setting
Security considerations
Consult with your security team before configuring the Image URL proxy
whitelist setting. The decision to bypass image proxy whitelist
protection can expose your users and domain to security risks if not
used with care.
In general, if you have a domain that needs authentication via cookie,
and if that domain is controlled by an administrator within your
organization and is completely trusted, then whitelisting that URL
should not expose your domain to image-based attacks.
Important: Disabling the image proxy is not recommended. This option is available to provide flexibility for administrators, but
disabling the image proxy can leave your users vulnerable to malicious
attacks.
Entering Image URL patterns
To maintain a whitelist of internal URLs that'll bypass proxy
protection, enter the image URL patterns in the Image URL proxy
whitelist setting. Matching URLs will bypass the image proxy.
A pattern can contain the scheme, the domain, and a path. The pattern
must always have a forward slash (/) present between the domain and
path. If the URL pattern specifies a scheme, then the scheme and the
domain must fully match. Otherwise, the domain can partially match the
URL suffix. For example, the pattern google.com matches
www.google.com, but not gle.com. The URL pattern can specify a
path that's matched against the path prefix.
Important: Enter your actual domain name as you enter the image URL pattern. Always include a trailing forward slash (/) after the
domain name.
Examples of Image URL patterns
The following patterns are examples only. The following patterns:
http://rule_fixed_scheme_domain.com/
rule_flex_scheme_domain.com/
rule_fixed_subpath.com/cgi-bin/
... will match the following URLs:
http://rule_fixed_scheme_domain.com/
http://rule_fixed_scheme_domain.com/test.jpg?foo=bar#frag
http://rule_fixed_scheme_domain.com
rule_flex_scheme_domain.com/
t.rule_flex_scheme_domain.com/test.jpg
http://t.rule_flex_scheme_domain.com/test.jpg
https://t.rule_flex_scheme_domain.com/test.jpg
http://rule_fixed_subpath.com/cgi-bin/
http://rule_fixed_subpath.com/cgi-bin/people
Note: The URL scheme (http://) is optional. If the scheme is omitted, the pattern can match any scheme, and allows partial matches
on the domain suffix.
Previewing the image URL patterns
Click Preview to see if the URLs match the image URL patterns
you've set. If the image URL matches a pattern, you'll see a
confirmation message. If the image URL does not match, an error
message appears.
Bharata has a great and detailed answer on this, but just wanted to add one addition that I identified with a similar issue.
We had a x-webkit-csp content security header that turned out to be the culprit.
Removing it and all worked through the image proxy.
Google's response was that x-webkit-csp is deprecated and to use the Content-Security-Policy header instead.
However this seems like a bug that an unsupported header throws a fatal error rather than simply ignoring it.
TL;DR: Make sure your server isn't blocking external connections (through AWS or .htaccess or some other firewall)!
I was having this problem too. I ran through every solution I could think of and every one I found online. Nothing fixed it.
Finally, I inspected the image in Gmail so that I could get the full CDN address Google creates for it. I tried to open that in a new tab and it failed. So I realized that Google wasn't able to grab the image.
In the end, I'd forgotten that I have the server locked down from all traffic except for my own (just a basic .htaccess IP deny). It's just a simple security layer I use while I'm in my development. Keep in mind that you might have it locked down within AWS or something more rugged like that.
I opened up all IPs for a minute, tested it, and sure enough it worked as expected. The old emails that were previously failing also worked, so it seems that Google tries to work their magic anytime the email is opened and they don't have the image saved. Once I closed the IP address again, the image continued to work whatever Google. I'm guessing once they write it to their CDN is remains there indefinitely.
So if you're certain that you've done everything else correctly, I would suggest making sure that the server is indeed open to the outside world so Google can process the image.
I had the same problem and I solved it by specifying the "https://" protocol in the "src" url of the img, otherwise by default "http" is prepended
I set the HSTS header on my site and i want to test that the different browsers (chrome, Firefox, IE, Opera) do enforce the header.
I set a trusted certificate, connect to the site and I can see the the header at the HTTP response. but i want to validate that the browser do enforce the protocol.
In Chrome it's easy and it works:
- I can query the site at chrome://net-internals/#hsts
- When trying to connect with HTTP i get 0kb response with status 307.
- If i change back self-signed cert i can't connect the site and there is no proceed option.
The other browsers behave differently, i can't query the HSTS list, the response status and size is different and when changing to self-signed cert (after first trusted connection) i do have proceed option.
So how can i validate that the protocol is enforced on each browser?
Although Chrome's ability to query the HSTS cache and see the fake 307 redirect is handy, you can just check whether HSTS is enforce.
HSTS offers you two options:
Automatically load HTTP resources over HTTPS
Prevent click through of cert errors.
You are concentrating on the second option, but why not use the first option as the test? Just load the site up with the HTTP and check whether it is redirected (i.e. loads HTTP URL and so is not using a HSTS rule) or if it just loads HTTPS URL immediately (i.e. is using HSTS).
So in Firefox for example open network tools click on "Persist logs" option (and let's do "Disable Cache" to avoid any confusion). Then go to a site which has an HSTS header over HTTP (e.g. http://stackoverflow.com) and you'll see a 301 redirect if this is your first visit:
Next time you go to it (after it has cached the HSTS header) it should go directly to the HTTPS URL even though you typed the HTTP URL in address bat:
If you've already been on stackoverflow.com then you can clear the HSTS cache to try this again.
Once you've confirmed that HSTS is being used or not, you can then investigate the click-through issue. Browsers should not allow click through when HSTS is in place, including for self-signed certs, but maybe there's a bug, or it's still cached your old cert in some places, or the HSTS policy has expired or something else...
I'm migrating my website from http to https (although it will still support access via http)
Currently all of my pages have accurate rel-canonical meta tags set in the HTML, but obviously they all point to the canonical http:// url.
Should I now be updating those to https:// too, or is it ok to leave them as http?
I'm wondering whether Google will penalise me, or start detecting duplicate content, if I start mixing them
Yes Google sees http and https as different sites so you should update them.
A redirect on the server might be sufficient in the short term but personally I would be looking to update the pages as soon as you can.
My site is using HTTPS only.
I allow using BBCodes to show images. Users are placing images like "https://imagehoster.net/img.png" and the imagehoster is using a redirect so the browser loads it via HTTP "http://imagehoster.net/img.png". This makes the browser showing annoying mixed content warnings. Is there a way to prevent this?
Short: NO
Long:
the have no really web server listening to ssl.
in fact, there is only a firewall/proxy which sends a http locate to the browser.
You can't intercept that request. even if you could, where to redirect to?
they don't provide a ssl server, because it takes to much resources for encryption or it takes to much traffic, because proxy#s can't cache.
An idea to solve that problem:
detect those links, download them and store a copy on your server.
replace the link. maybe you need only to store a preview. if the click on it redirect to the original link on a new browser window.
While adding a hyperlink to another site (which has SSL), the site documentation sometimes say to link to the http:// link instead of the https:// (e.g. Play store, which is a site that uses SSL but it does not tell you to link to https; instead, it says to link to http). They do not matter (as they function normally), but would there be a reason to link to the http:// instead the https://?
Maybe they don't want extra encryption and lowering down the site speed as SSL may decrease performance somewhat.
If users are downloading large, public files, there may be a system burden to encrypt these each time.
Some browsers may not support SSL.
You will probably want the home page accessible via HTTP, so that users don't have to remember to type https to get to it.
Your specific portion of page needs secure http(https) not whole site.
Your site is indexed mainly on http on Search engines.