Microsoft's confirmation of a WSUS sync issue – failing to receive updates (May 11, 2021)

[German]As of May 11, 2021, administrators found that no Windows updates were found in the Windows Server Update Service (WSUS). After a few hours, the problem resolved. Now Microsoft has confirmed the problem nor provided some hints.

I had reported the issue in the blog post WSUS does not receive updates (May 11, 2021) after a blog reader pointed it out to me. WSUS showed successful synchronization, but little came except Defender updates – most notably, the Windows updates released on Patchday (May 11, 2021) did not show up. In the blog post linked above, there were references to timing problems and that Microsoft wanted to fix the problem. Did happen then, as readers confirm.

Microsoft explains the glitch

Microsoft has now added a new entry in the status area of Windows 10, in which they once again address the issue. Redmond confirms that when checking for updates in Windows Server Update Services (WSUS) or Microsoft Endpoint Configuration Manager and managed devices that connect to these servers, the security update for the Windows client may not be available or may not have been offered. This may also affect the Security Only and Internet Explorer cumulative rollups on the Windows platforms. Virtually all supported Windows versions were affected, as the following list shows.

• Client: Windows 10, version 20H2; Windows 10, version 2004; Windows 10, version 1909; Windows 10, version 1809; Windows 10 Enterprise LTSC 2019; Windows 10, version 1803; Windows 10 Enterprise LTSC 2016; Windows 10, version 1607; Windows 10 Enterprise 2015 LTSB; Windows 8.1; Windows 7 SP1.
• Server:  Windows Server, version 20H2; Windows Server, version 2004; Windows Server, version 1909; Windows Server, version 1809; Windows Server 2019; Windows Server, version 1803; Windows Server 2016; Windows Server 2012 R2; Windows Server 2012; Windows Server 2008 R2 SP1; Windows Server 2008 SP2.

In the meantime, however, the problem has probably been solved. Microsoft writes that this problem has now been fixed on the "service side" (i.e. on the Microsoft servers that provide the updates). According to my information, wrong time zone information was set for the release of the packages, so that the updates were not released for distribution. The updates should be available in WSUS and co. in the meantime.

Note: If a sync cycle is initiated and the updates are still not offered, cyclic checks should be made to see if updates arrive. By now, however, the fix should have been distributed to all servers in all regions worldwide. (via)

Some Insights

A German blog reader has left a comment with further explanations under my German edition of this blog post. Here is the translation.

The ServerSyncWebService, with which Microsoft provides the updates, works similar to a WSUS, if it is parent of another WSUS.

You can not only set or deny a release for an update, but also set a time (TTLG, time to go live) for it.
This is done by Microsoft (the patches themselves are stored in the database beforehand and can be downloaded by members of special test groups, but not by the general public.

As said, all standard behavior of the WSUS API.

Due to the Reasonable Disclosure Policy, patches are supposed to be made available at the same time worldwide, so as not to give certain parts of the world an unfair advantage (be it simply updating earlier or be it analyzing the patches to develop for exploits).

Microsoft has long set this time at 10 a.m. local time at HQ in Seattle (which is the U.S. Pacific time zone PST/PDT, i.e. GMT-7) on Patchday (usually Tuesday of B-week). Due to the different daylight saving time rules on both continents (first weekend of April and last weekend of March, respectively), this is often 9, sometimes 10 hours off.

The team, which adds the security updates, got the time for the TTGL exactly 7 hours wrong and so Microsoft's update server automatically made the updates available at 5pm Seattle Time.

The second linked tweet from John Storbeck showed that Microsoft tried to correct the release date directly in the database of their update server. Since such a SQL update statement has much more potential to cause damage than the several times secured UI, it needed a SQL guru with the necessary experience (and access rights) so that this was not done exactly 10:01 am, but sometime during the night.