Ouch: Let's encrypt withdraws 3 million certificates

[German]A brief message at the end of the day here in German: Let's encrypt had found a bug in the process of issuing certificates. So they now have to withdraw 3 million issued TLS certificates.


Reaching for the stars: One billion certificates issued

I had already noticed it a few days ago, Let's encrypt issued 1 billion TSL certificates. I deliberately didn't have it in my blog – but it was briefly discussed here today in German blog reader Ralf's comment. The article of the Let's Encrypt makers is available here.

Grandiose success, and we can only congratulate. I also started with the free Let's Encrypt TLS certificates here in the blog, but since last year I have a paid certificate that lasts for 12 months.

3 million certificates invalid

The above success story is the reach for the stars. But now back to the lowlands of practice. I just read in Golem that Let's Encrypt has to revoke 3 million TLS certificates due to a mistake.

The error causes that the check of the CAA DNS records can not be performed correctly. The details are disclosed in the Let's Encrypt community:


On 2020-02-29 UTC, Let's Encrypt found a bug in our CAA code. Our CA software, Boulder, checks for CAA records at the same time it validates a subscriber's control of a domain name. Most subscribers issue a certificate immediately after domain control validation, but we consider a validation good for 30 days. That means in some cases we need to check CAA records a second time, just before issuance. Specifically, we have to check CAA within 8 hours prior to issuance (per BRs §, so any domain name that was validated more than 8 hours ago requires rechecking.

The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let's Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let's Encrypt.

We confirmed the bug at 2020-02-29 03:08 UTC, and halted issuance at 03:10. We deployed a fix at 05:22 UTC and then re-enabled issuance.

Our preliminary investigation suggests the bug was introduced on 2019-07-25. We will conduct a more detailed investigation and provide a postmortem when it is complete.

In brief: The verification of CAA DNS records is not properly checked during certificate issuance. This makes it possible to issue a certificate for a domain with a validity of up to x+30 days, even if someone later installs CAA records for this domain name that prohibit issuance by Let's Encrypt. The bug exists since July 25, 2019.

Affected users have been informed by mail and must now have a new certificate issued immediately. On the website here, the URL of a website to be checked can be entered. Then it will be checked whether they are affected by the withdrawn certificates. 

Hanno Böck has published a small script on GitHub that checks if you need to renew your certificate.

Cookies helps to fund this blog: Cookie settings

This entry was posted in Security and tagged . Bookmark the permalink.

One Response to Ouch: Let's encrypt withdraws 3 million certificates

  1. Pingback: Secure websites with SSL and HTTPS - UKBSS LTD

Leave a Reply

Your email address will not be published. Required fields are marked *