We've recently gotten a 'real' ssl certificate from godaddy to cover our domain (*.domain.com) and have installed it in several places where needed for email (imap/starttls and etc) and web. This works great, seems ok according to various online TLS certificate checkers, and I get the green lock when testing using my own browsers and such.
I have a customer however that uses our web mail system now secured with ssl. I myself and many others use it and get the green lock. But, whenever any station at the customer tries using it, they get a broken lock and 'your connection is not private'. The actual error displayed below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate Authority - G2". And it gets worse - whenever I go to the location and use my own laptop, the very one that 'works' when at my office, I ALSO get the error. AND EVEN WORSE - when I connect to my cell phone provided hotspot, the error goes away!
As weird as this all sounds, I got it nailed down to one device - they have a Cisco/Meraki MX64W as their internet gateway - and when I remove that device from the chain and go 'straight' out to the internet, suddenly, the certificate problem goes away entirely.
How is this possible? Can anyone comment on these devices and tell me what might be going on here?
It's been compromised and its being used for MITM? Or has some sort of TLS inspection capability built in which is essentially MITM, and which is enabled? Or some kind of content filtering capability which amounts to the same thing?
Thu, Mar 26, 2015 at 03:38:55PM -0700, Mike wrote:
I have a customer however that uses our web mail system now secured
with ssl. I myself and many others use it and get the green lock. But,
whenever any station at the customer tries using it, they get a broken
lock and 'your connection is not private'. The actual error displayed
below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate
Authority - G2". And it gets worse - whenever I go to the location and
use my own laptop, the very one that 'works' when at my office, I ALSO
get the error. AND EVEN WORSE - when I connect to my cell phone provided
hotspot, the error goes away!
As weird as this all sounds, I got it nailed down to one device -
they have a Cisco/Meraki MX64W as their internet gateway - and when I
remove that device from the chain and go 'straight' out to the internet,
suddenly, the certificate problem goes away entirely.
How is this possible? Can anyone comment on these devices and tell
me what might be going on here?
Sounds like deep packet inspection (DPI) with SSL MITM. Reading https://meraki.cisco.com/lib/pdf/meraki_datasheet_mx.pdf
makes me believe that this device can do that. Look for it's
configuration, DPI for HTTPS must be active.
I have found they cause issues with Linux firewalls if the merakis are not configured "correctly".
Meraki Access Points do content inspections which I have found can cause produce symptoms similar to yours, although I have not experienced what you are describing. Since the MX64W is both an Access Point & security gateway, it has some additional content inspection/intelligence for it's security appliance role on top of the functions it performs as an access point, the same functions which are found in Meraki standalone access points as well.
I am not sure what the specifics are as I do not use Meraki security appliances but it is worth checking. I have found with Meraki that items in the control panel/dashboard are not always labeled the best so I have found it is usually worth putting in a ticket with them and/or a call to them to see what they think (1-888-490-0918).
I'd like to thank everyone for their kind responses. One person who responded off list and bothered to look at the returned certificates pointed out, and correctly it seems, that my original setup was missing an intermediate certificate. The site was returning 'valid ssl' and all browsers got the green lock and offsite ssl tests came back ok, but apparently the missing intermediate means it would have had to have been fetched and that was the part that was failing at the customer site. Once I put the intermediate certificate in there, the customer site was able to access https without fail. I have not had an opportunity yet to examine in detail the config of the meraki router there but it's either a routing problem or a DPI problem. If I get an answer I'll post again with my results.
Thanks. Previously while diagnosing this however, I used some others similar and they all were saying I was ok. For example, https://www.ssllabs.com/ssltest/analyze.html and one other I forget now. I am surprised this problem was not being pointed out.
I believe the SSLLabs Analyzer should have pointed out an "Extra Download" in the cert chain. That was the hint that there was an intermediate cert that a client would have to go find on it's own because it wasn't included with your server cert.
We ran into this when OS X changed how it handles SSL a few years
back, our captive portal was presenting a splash page in place of
Thawte OCSP and crashing the SSL keychain process. The work-around
was either to respond with a TCP RST for these requests or to allow
them through.
I went back to Frank's list and did some additional testing. I have a different server which was set up the same way as the previous one discussed, and I thought I would use the above tools and see if my problem would have been identified by any of them. I am sorry to report, no, none of these either caught the problem either. Although I still do not fully understand the dependencies involved, it seems that if my server was failing to supply the full certificate chain, and the browser was compensating for it by (attempting?) to load the missing certificate from elsewhere, and this Meraki router was somehow able to confound that process, that would be an issue worthy of exploring more. I certainly don't blame these ssl check sites but clearly theres more checks needed.
The Qualsys site (https://www.ssllabs.com/ssltest/analyze.html) will report whether or not the server supplied the intermediate cert. But I agree with you that the other tools should make a bigger deal about it if the server doesn't supply it.
FWIW, it's been the CW to do this for some time now, as there are systems like the one you've run into that were designed before intermediate certs were commonplace, and don't know how to handle them.
I've also experienced situations where an enterprise purchases a DV certificate to be used on an offline system, and while that system has access to the "root" CA certs, it cannot retrieve the intermediate cert. Having the end system supply the intermediate cert as well solves this issue.
The method of supplying the intermediate cert is simple, just append the intermediate certificate to the end of the file with your server certificate (the .crt file). Any reasonably modern software will handle that transparently, and provide the intermediate cert along with the server cert when doing its business.
I have a new godaddy certificate for *.mydomain.com, and that is what I installed. However, the certificate chain I supplied was missing some intermediate godaddy certificate. Originally, it appeared I was missing 'gdig2.crt', and once installed, that fixed some clients including the ones behind the meraki router. But then there were also some older clients this did not fix (a macos 10.4 something for example). So I went back and installed gd_bundle-g2-g1.crt in it's place, and that seems to have finally done it.
I apologize for the diminishing lack of operational content. It just seems that these ssl tests should be tightened up and perhaps some additional tools deployed out there to help us less knowledgeable folks 'get it right'.
That's something I suspected at first, it but discounted when your said your laptop also failed at the site.
The first intermediate you installed took care of anything with the newer root certificates installed.
But for your older 10.4 Mac clients (which presumably haven't had a root certificate bundle update in a while) that wasn't enough - the new root needed to be provided since from their perspective it's an intermediate.
Back in the good aulde days all of the SSL certs one might buy were
signed directly by the CA, but now more often than not there are
intermediate certs, and a valid cert needs to be accompanied by all of
the intermediate certs between it and the CA.
What makes debugging hard is that browsers try to be helpful. If a
server doesn't provide the intermediate certs, but the browser happens
to have them in its cache from some other site, well, close enough and
the SSL works. But if some other browser doesn't happen to have them,
you lose.
So if your SSL is flaky, check those intermediate certs first.
With all this resolved, I'll note that I just reviewed
draft-ietf-tls-sslv3-diediedie, which is in IETF Last Call prior to publication as an RFC. It deprecates the use of any version of SSL in favour of TLS 1.2 in the clientHello negotiations.
