[German]As of September 30, 2021, some root certificates that Let's-Encrypt used to sign user certificates expired. This meant that certain devices or applications could no longer access websites or mail servers. I have seen cases with iOS 14/15 and macOS, as well as problems with Internet Information Server (IIS) and Microsoft Exchange. In addition, the Sophos UTM Firewall Application is causing problems because it also uses an affected Let's Encrypt certificate. However, the problems can be solved easily – here is a short overview.
Advertising
The Let's-Encrypt certificate problem
As of today, September 30, 2021, some root certificates used by Let's Encrypt to sign client certificates will lose their validity (expiration of Intermediate R3 on 9/29/2021 at 19:21:40 GMT – the DST Root CA X3 expires on 9/30/2021 14:01:15 GMT). Clients that only know the old root certificates will not be able to verify Let's Encrypt server certificates after that. I had pointed this out in the blog post Sept. 30, 2021: Will we see trouble with old Let's Encrypt certificates? However, it was unclear whether this would have an impact – it was assumed that this would only affect very old devices that no longer receive updates (e.g. Android < v2.3.6). In the meantime, however, it is clear from reader feedback that devices with iOS 14 or iOS 15, macOS etc. may also be affected.
Problems with Exchange and IIS
I have received a few comments reporting problems with Exchange Server as well as Windows Internet Information Server (IIS). The problem was that the server in question preferred the older Let's Encrypt certificate. In this case, restarting the servers in question fixed the certificate chain – as can be seen from the following.
Windows IIS: Issues with iOS
Windows Internet Information Server (IIS) seems to have had problems with iOS devices. On German site heise there is this German comment, which contains some hints.
Issues mit Let's Encrypt certificate, but that only occurred with macOS and iOS, when retrieved from a Windows server IIS.
On all Windows devices there were no problems, with any browser.
On macOS, the problem also occurred with Firefox.Workaround was:
-delete the expired X3 certificate in the Windows server certificate store.
-issue all Let's-Encrypt certificates NEW
-restart IIS (only for security)After that the Apple devices had no problems, a restart there was not necessary. Why this only occurred on macOS and IOS is not entirely clear.
Other comments like this German one within my blog confirm the issues with iOS devices. Also in this German comment on heise someone complains about issues with IIS and Let's Encrypt certificates. Martin Bene wrote within this comment here in my blog:
There are issues with IIS; the certificates are actually OK, but when building the certificate chain it sends, it prefers the old and now expired R3 intermediate certificate.
It keeps sending the expired intermediate certificate even after the actual expire date until the server is rebooted; this breaks Clients that don't provide intermediate certificates themselves (like iOS). Other clients (Windows) continue working just fine.
* Renewing the certificates on the server causes the chains to be rebuild and fixes the issue
* rebooting the server causes the chains to be rebuilt and also fixes the issue.
* just issuing an iisreset does NOT fix the issue
This confirms the problem that in many cases an updated Let's Encrypt certificate does not take effect. Only a restart of the Windows server causes a new certificate to be included in the certificate chain. Unfortunately, resetting the IIS does not help. After that, the iOS devices were able to communicate again.
Advertising
Issues with Exchange
Also in this German comment Markus reports problems in connection with the accessibility of an Exchange server due to certificate problems. Here is his comment:
Solution on Exchange server renew the certificate ! you should work with the win-acme solution from Frankys web!
Here I can give again the hint which reached me via Facebook: In my case an Exchange Server restart helped. After the restart the certificate chain is displayed correctly on the server.
Sophos UTM affected
In my German blog, several users reported issues with access from iOS devices in conjunction with Sophos UTM (firewall appliance). Blog reader Michael J. wrote in this German comment:
it really looks like there are problems with the internal mail app under the current iOS 15 release. Here I get a message displayed that the certificate has expired, although an expiration date is displayed in the future.
Have checked the certificate directly on the server (win-acme), as well as reverse proxy (Sophos UTM with LE), all OK.
Can anyone verify/confirm this?
A short time later I also got feedback on Facebook that iPhones with iOS 14 could not use communication with mail servers either. Michael adds in a later comment:
I was able to identify the problem. In the Sophos UTM, in addition to the valid root CAs, there were the two certificates that had already expired. I deleted them and the problem with the chain was solved. Might be quite useful if you are troubleshooting.
According to this German comment, these can be found and deleted under Webserver Protection -> Certificate Management -> Certificate Authority. The problem may have to do with the issue mentioned above of new certificates being disregarded in the certificate chain until the underlying server is rebooted.
On Facebook I have received another feedback: Is the same on Mac (BigSure), the root certificate (R3) expired yesterday. On Windows the certificates are valid.
It was due to the Exchange and the Sophos UTM. On the UTM the old CA's had to be deleted, then everything was displayed correctly again in the browser. Only Outlook on the Mac was still complaining. Here, the expired root certificate had to be deleted from the certificate store, then the Let's Encrypt certificate renewed and an IIS reset performed. After that the Mac connects without errors again….
(Source: Sophos)
Addendum: Sophos has release an Advisory: Let's Encrypt Root Certificate Expiry including the image shown above how to fix that issue (thanks to my reader for the tip). Maybe the collected hints will be helpful for someone.
Advertising
I found that SSL inspection on Sophos UTM also failed on HTTPS sites with Let's Encrypt certificates.
I had to disable the "Digital Signature Trust Co. DST Root CA X3" and "Internet Security Research Group ISRG Root X1" Global Verification CAs and upload a new "Internet Security Research Group ISRG Root X1" root certificate as a local verification CA under Web Protection > Filtering Options >HTTPs CAs and restart Web Protection to be able to access those sites.
Delete Root CA and renew worked for our Sophos UTM admin URL and all WAF protected sites. Thanks!
Even without the X3 CA installed and after a renew of the certificate today, the old expired chain is still send from the UTM to clients…
I have an Exchange 2016 server running on Server 2012R2 running the latest CU of Exchange. I have it setup to permit incoming IMAP. It had the same issue as discussed when the DST cert expired. I rebooted it as per the instructions here. After it came back up it still had the same problem. However, I checked it 3 hours later – and the problem disappeared. Obviously the reboot triggered a rebuild of the certificate chains – but just as obviously, that rebuild DIDN'T happen immediately. Or the IIS cert cache didn't get updated or something else. Anyway, just a note to let people know to be patient after rebooting a Windows server since it may take a while for it to get around to doing it's housekeeping.
The previous blog suggested that Win XP equal to or later than SP3 would be OK, as long as the automatic certificate update was checked; I've found that I do have this checked, but my SP3 is failing due to the certificate expiry.
Let's Encrypt website offers a download of the new X1 cert, but it downloads without specifying what the filename is, nor where it downloaded to. I attempt to Import the new certificate, and the Import dialogue does not know what to import, nor where to find it….