I've written an intercepting proxy in Python 3 which uses a man-in-the-middle "attack" technique to be able to inspect and modify pages coming through it on the fly. Part of the process of "installing" or setting up the proxy involves generating a "root" certificate which is to be installed in the browser and every time a new domain is hit via HTTPS through the proxy, the proxy generates a new site certificate on-the-fly (and caches all certificates generated to disk so it doesn't have to re-generate certificates for domains for which certificates have already been generated) signed by the root certificate and uses the site certificate to communicate with the browser. (And, of course, the proxy forges its own HTTPS connection to the remote server. The proxy also checks the validity of the server certificate if you're curious.)
Well, it works great with the browser surf. (And, this might be relevant -- as of a few versions back, at least, surf didn't check/enforce certificate validity. I can't attest to whether that's the case for more recent versions.) But, Firefox gives a SEC_ERROR_REUSED_ISSUER_AND_SERIAL error on the second (and all later) HTTPS request(s) made through the proxy and Chromium (I haven't tested with Chrome proper) gives NET::ERR_CERT_COMMON_NAME_INVALID on every HTTPS request. These obviously present a major problem when trying to browse through my intercepting proxy.
The SSL library I'm using is pyOpenSSL 0.14 if that makes any difference.
Regarding Firefox's SEC_ERROR_REUSED_ISSUER_AND_SERIAL error, I'm pretty sure I'm not reusing serial numbers. (If anybody wants to check my work, that would be pretty rad: cert.py - note the "crt.set_serial_number(getrandbits(20 * 8))" on line 168.) The root certificate issuer of course doesn't change, but that wouldn't be expected to change, right? I'm not sure what exactly is meant by "issuer" in the error message if not the root certificate issuer.
Also, Firefox's "view certificate" dialog displays completely different serial numbers for different certificates generated by the proxy. (As an example, I've got one generated for www.google.com with a serial number of 00:BF:7D:34:35:15:83:3A:6E:9B:59:49:A8:CC:88:01:BA:BE:23:A7:AD and another generated for www.reddit.com with a serial number of 78:51:04:48:4B:BC:E3:96:47:AC:DA:D4:50:EF:2B:21:88:99:AC:8C .) So, I'm not really sure what Firefox is complaining about exactly.
My proxy reuses the private key (and thus public key/modulus) for all certificates it creates on the fly. I came to suspect this was what Firefox was balking about and tried changing the code to generate a new key pair for every certificate the proxy creates on the fly. That didn't solve the problem in Firefox. I still get the same error message. I have yet to test whether it solves the Chromium issue.
Regarding Chromium's NET::ERR_CERT_COMMON_NAME_INVALID error, the common name for site certificate is just supposed to be the domain, right? I shouldn't be including a port number or anything, right? (Again, if anybody would like to check my work, see cert.py .) If it helps any, my intercepting proxy isn't using any wildcards in the certificate common names or anything. Every certificate generated is for one specific fqdn.
I'm quite certain making this work without making Firefox or Chrome (or Chromium or IE etc) balk is possible. A company I used to work for purchased and set up a man-in-them-middling proxy through which all traffic from within the corporate network to the internet had to pass. The PC administrators at said company installed a self-signed certificate as a certificate authority in every browser on every company-owned computer used by the employees and the result never produced any errors like the ones Firefox and Chromium have been giving me for the certificates my own intercepting proxy software produces. It's possible the PC administrators tweaked some about:config settings in Firefox to make this all work or something, but I kindof doubt it.
To be fair, the proxy used at this company was either network or transport layer, not application layer like mine. But I'd expect the same can be accomplished in an application-layer HTTP(s) proxy.
Edit: I've tried setting the subjectAltName as suggested by brain99. Following is the line I added in the location brain99 suggested:
r.add_extensions([crypto.X509Extension(b"subjectAltName", False, b"DNS:" + cn.encode("UTF-8"))])
I'm still getting SEC_ERROR_REUSED_ISSUER_AND_SERIAL from Firefox (on the second and subsequent HTTPS requests and I'm getting ERR_SSL_SERVER_CERT_BAD_FORMAT from Chromium.
Here are a couple of certificates generated by the proxy:
google.com: https://pastebin.com/YNr4zfZu
stackoverflow.com: https://pastebin.com/veT8sXZ4
I noticed you only set the CN in your X509Req. Both Chrome and Firefox require the subjectAltName extension to be present; see for example this Chrome help page or this Mozilla wiki page discussing CA required or recommended practices. To quote from the Mozilla wiki:
Some CAs mistakenly believe that one primary DNS name should go into
the Subject Common Name and all the others into the SAN.
According to the CA/Browser Forum Baseline Requirements:
BR #9.2.1 (section 7.1.4.2.1 in BR version 1.3), Subject Alternative
Name Extension
Required/Optional: Required
Contents: This extension MUST contain at least one entry. Each entry MUST be either a dNSName containing the Fully-Qualified Domain Name or an iPAddress containing the IP address of a server.
You should be able to do this easily with pyOpenSSL:
if not os.path.exists(path):
r = crypto.X509Req()
r.get_subject().CN = cn
r.add_extensions([crypto.X509Extension("subjectAltName", False, "DNS:" + cn])
r.set_pubkey(key)
r.sign(key, "sha1")
If this does not solve the issue, or if it only partially solves it, please post one or two example certificates that exhibit the problem.
Aside from this, I also noticed you sign using SHA1. Note that certificates signed with SHA1 have been deprecated in several major browsers, so I would suggest switching to SHA-256.
r.sign(key, "sha256")
Related
The domain localhost is treated as a secure origin by default, which is very handy when testing webapps locally.
It let's you use features like getUserMedia without having to use the https protocol.
Because crostini uses the domain linuxhost instead of localhost, it isn't treated as a secure origin.
There is a chrome flag, chrome://flags/#unsafely-treat-insecure-origin-as-secure, that lets you set new domains to be treated as secure, but I've tried all of the following, and everytime, when I restart chrome, the input is empty again.
linuxhost
http://linuxhost
http://linuxhost:3000
http://linuxhost:3000/
plus all the other combinations of http://, linuxhost, :3000, and /.
There are some threads about treating linux.test as secure and changing linuxhost to linux.test but those are pretty recent and don't work yet.
Has anyone managed to get that flag to work on chrome os.
I've seen other answers that show how to set this flag, but only via the command line, not through the chrome UI.
I'm on the dev channel with crostini working, on version 69.0.3473.0
The google's certificate transparency project has been in place for some time, google chrome and mozilla firefox have both claimed to have joined the project, but how do I test if the browser actually suports certificate transparency and the three ways of delivery of SCT?
One of the easiest ways to test whether a browser is checking certificate transparency is to try a known bad site, such as https://invalid-expected-sct.badssl.com. Using this address, Chrome 69 will say the site is insecure, but Safari 12.0 which doesn't perform certificate transparency will let it through.
Chrome's policy can be found at https://github.com/chromium/ct-policy/blob/master/ct_policy.md
Apple are in the process of enforcing certificate transparency with I believe the plan being to roll it out in iOS 12.1.1 and macOS 10.14.2. Their policy can be found at https://support.apple.com/en-us/HT205280
Firefox 63.0.1 doesn't seem to support certificate transparency either although support is built into Firefox I believe it is currently not enforced until some other issues are resolved.
In terms of trying to test the three methods of delivery there is a research project at https://www.ida.liu.se/~nikca89/papers/pam18.html with code available that pulls SCTs for a given list of domains so you should be able to use that to check all 3 ways. To get it working you create a file top-1m.csv with entries for each domain on separate lines prefixed with an ignored numeric value and execute the main function in FirstTestCase. Alternatively you could look at the Conscrypt project although that is more work.
Does using Chrome v.63 force use of https?
I am running Apache 2.4.27 on a Windows 10 desktop as a sandbox where I can experiment and do some tutorials. I have a virtual host setup called www.tutorial.dev with an alias of tutorial.dev. In the Windows 10 hosts file I have set up www.tutorial.dev and tutorial.dev to point to localhost.
As of yesterday the url http://tutorial.dev/Bootstrap4FromScratch/ was working normally. In this case providing a directory listing as a jump off point into various examples and exercises. Today, when I type in the url Chrome changes it to https and I get a connection refused message.
I understand the connection refused message. There are no certificates setup.
The only change I can find is that Chrome changed from v.62.x to v.63.x. What in Chrome 63 could be forcing http to https?
I don't have this problem with MS Edge. I tested another similar configuration on a different machine that was in the process of downloading Chrome 63.x. It already had 62.x installed. It worked until the 63.x upgrade was complete then the same problem occurred.
Additional information: If I use http://localhost to bring up the index.html or version.php in the htdocs directory the switch from http to https does not happen. The virtual host www.tutorial.dev resides in another directory outside of htdocs.
If this has been asked and answered please point me to the question/answer thread.
Thanks in advance,
Barry
Google owns the .dev TLD and with Chrome 63 they are forcing HTTPS on all requests to anything.dev
I went through my local dev setup and replaced all references to .dev with .local, works fine now.
Your other option is to use Firefox for local development. .dev now triggers https in FF (since before FF61) with a workaround
Edit (asside):
I have switched to using .localhost for dev as browsers allow navigator.geolocation.getCurrentPosition() (blocked if site is not HTTPS).
Google Chrome 63 update, out December 2017, places .dev domains in the preloaded HSTS list with a rule enforcing HTTPS, no workarounds.
{ "name": "dev", "include_subdomains": true, "mode": "force-https" }
The "only" way is to switch .dev with something else, like .localhost
IETF states a few reserved TLDs for development:
TLDs for Testing, & Documentation Examples
There is a need for top level domain (TLD) names that can be used
for creating names which, without fear of conflicts with current or
future actual TLD names in the global DNS, can be used for private
testing of existing DNS related code, examples in documentation, DNS
related experimentation, invalid DNS names, or other similar uses.
For example, without guidance, a site might set up some local
additional unused top level domains for testing of its local DNS code
and configuration. Later, these TLDs might come into actual use on
the global Internet. As a result, local attempts to reference the
real data in these zones could be thwarted by the local test
versions. Or test or example code might be written that accesses a
TLD that is in use with the thought that the test code would only be
run in a restricted testbed net or the example never actually run.
Later, the test code could escape from the testbed or the example be
actually coded and run on the Internet. Depending on the nature of
the test or example, it might be best for it to be referencing a TLD
permanently reserved for such purposes.
To safely satisfy these needs, four domain names are reserved as
listed and described below.
.test
.example
.invalid
.localhost
".test" is recommended for use in testing of current or new DNS
related code.
".example" is recommended for use in documentation or as examples.
".invalid" is intended for use in online construction of domain
names that are sure to be invalid and which it is obvious at a
glance are invalid.
The ".localhost" TLD has traditionally been statically defined in
host DNS implementations as having an A record pointing to the
loop back IP address and is reserved for such use. Any other use
would conflict with widely deployed code which assumes this use.
PS: .foo is also in the preloaded HSTS list
Thanks everyone for the advice. I ended up going with .tst for now. I have a feeling I'l be switching over (forced?) to .localhost at some point. But for now .tst is less typing.
I have followed some tutorials in adding a self-cert to my local environment, mainly this one Getting Chrome to accept self-signed localhost certificate. I've added my certificate to the keychain and it is now marked as trusted for all users. But I'm still getting the error below. I've never tried to do this before and unsure now of the problem and how to fix it. Any advice would be great
Update
Steffen's answer helped me fixed this so I have awarded the points to him. I used this link for reference https://serversforhackers.com/video/self-signed-ssl-certificates-for-development
While in theory a certificate for *.dev will match test.dev in practice most browsers do not allow wildcards on the second level, i.e. *.com or *.org. It does not help that dev is currently not a public top level domain.
I recommend that you instead issue yourself the certificate without the wildcard (test.dev) or use an additional level for your hostnames.
On an intranet site, let's say I want to link to a file on a share using UNC, at:
\\servername\foldername\filename.rtf
It seems the correct way to do this is with markup like this:
filename.rtf
That's five slashes - two for the protocol, one to indicate the root of the file system, then two more to indicate the start of the server name.
This works fine in IE7, but in Firefox 3.6 it will only work if the html is from a local file. I can't get it to work when the file comes from a web server. The link is "dead" - clicking on it does nothing.
Is there a workaround for this in Firefox? Those two browsers should be all I need to worry about for now.
Since this is obviously a feature of Firefox, not a bug, can someone explain what the benefit is to preventing this type of link?
This question has been asked at least twice before, but I was unable to find those posts before posting my own (sorry):
Open a direct file on the hard drive from firefox (file:///)
Firefox Links to local or network pages do not work
Here is a summary of answers from all three posts:
Use WebDAV — this is the best solution for me, although much more involved than I had anticipated.
Use http:// instead of file:///// — this will serve up a copy of the document that the user cannot edit and save.
Edit user.js on the client as described here — this worked for me in Firefox 3.6.15, but without access to client machines, it's not a solution.
In Firefox, use about:config, change the Security.fileuri.strict_origin_policy setting to false — this doesn't work for me in 3.6.15. Other users on [SO] have also reported that it doesn't work.
Use the locallinks Firefox extension — this sets the Security.fileuri.strict_origin_policy to true for you, and appears to have no other effect.
Read the file server-side and send it as the response — this presents the same problem as simply configuring your web server to use http://.
Browsers like Firefox refuse to open the file:// link when the parent HTML page itself is served using a different protocol like http://.
Your best bet is to configure your webserver to provide the network mapped file as a web resource so that it can be accessed by http:// from the same server instead of by file://.
Since it's unclear which webserver you're using, I can't go in detail as to how to achieve this.
In Firefox to Open File:\\\\\yourFileServer\docs\doc.txt for example you need to turn on some options in Firefox configuration:
user_pref("capability.policy.policynames", "localfilelinks");
user_pref("capability.policy.localfilelinks.sites", "http://yourServer1.companyname.com http://yourServer2.companyname.com");
user_pref("capability.policy.localfilelinks.checkloaduri.enabled", "allAccess");
As it turns out, I was unaware that Firefox had this limitation/feature. I can sympathize with the feature, as it prevents a user from unwittingly accessing the local file system. Fortunately, there are useful alternatives that can provide a similar user experience while sticking to the HTTP protocol.
One alternative to accessing content via UNC paths is to publish your content using the WebDAV protocol. Some content managements systems, such as MS SharePoint, use WebDAV to provide access to documents and pages. As far as the end-user experience is concerned, it looks and feels just like accessing network files with a UNC path; however, all file interactions are performed over HTTP.
It might require a modest change in your file access philosophy, so I suggest you read about the WebDAV protocol, configuration, and permission management as it relates to your specific server technology.
Here are a few links that may be helpful if you are interested in learning more about configuring and using WebDAV on a few leading HTTP servers:
Apache Module mod_dav
IIS 7.0 WebDAV Extension
Configuring WebDAV Server in IIS 7, 6, 5
Add your own policy, open configuration "about:config" in the location bar and add three new entries:
capability.policy.policynames MyPolicy
capability.policy.MyPolicy.sites http://localhost
capability.policy.MyPolicy.checkloaduri.enabled allAccess
Replace http://localhost with your website.
Works with Firefox 70.0.
I don't know if this will work, but give it a shot! Old article, but potentially still useful.
http://www.techlifeweb.com/firefox/2006/07/how-to-open-file-links-in-firefox-15.html