[German]The timely installation of security updates is a building block to protect Windows systems from the cyber threats of this time. Microsoft therefore once again advertises to its clientele that it implements or achieves a "first-class monthly Windows patch efficiency". Here, in principle, one would nod immediately if it weren't for the teeny-tiny flaw of often unreliable updates, which, unfortunately, are recurrently notable for significant collateral damage. Here's a blog post that tries to illuminate the two different worlds between Microsoft's own view and the user community's experience.
Advertising
Touts for fast monthly Windows updates
On the so-called Patchday (2nd Tuesday at the moment), Microsoft releases its security updates for various products, including Microsoft Windows. And in the weeks before that, yes, there are the optional preview updates, which are actually used for testing, to identify problems in advance and eliminate them until Patchday by Microsoft. Furthermore, updates are installed automatically under Windows 10 (except for managed systems, where administrators release the packages). Actually, ideal conditions for keeping the Windows systems up to date with the latest patches – I thought.
The tweet above from Gabe Frost, Principal Lead Program Manager der Media Platform Group in the Operating Systems Division at Microsoft, dated July 22, 2021, but had passed me by. I first came across the topic over the weekend via this tweet from Microsoft employee @ariaupdated, who helps IT administrators manage Windows updates with the Windows Insider Program for Business, Windows Update for Business (WUfB) and WSUS. Gabe Frost has published the Techcommunity post Achieving world-class Windows monthly patching efficiency. There he restates Microsoft's ideas in this regard.
The article is aimed at those responsible for patch management in companies. Right at the beginning of the Techcommunity post, he asks rhetorically if readers have "the right approach" to achieving first-class protection for the devices you manage? For Microsoft's management, the key to protecting an environment's devices from today's threats is to patch them as quickly as possible. Every second Tuesday of the month – also known as Update Tuesday, Patch Tuesday or "B" release – Microsoft does, after all, release security updates for all Windows endpoints. These updates provide both home users and IT administrators with the necessary security updates to protect their devices before unpatched vulnerabilities can potentially be exploited.
Patching is too slow
In the article, Frost describes Microsoft's experience working intensively with customers. Companies have been able to dramatically improve their results by adopting a "cloud cadence" mentality and changing their approach to monthly maintenance, he says. By "cloud cadence," Microsoft means streamlining the review and approval processes to get security updates to devices quickly, before malicious actors exploit vulnerabilities. This approach, he said, is becoming increasingly important as companies respond and adapt to the challenges of remote working and business continuity in general. In the blog post, Frost then defined three update approaches to compare their monthly update adoption results:
Advertising
- Devices managed via Microsoft Update: These are devices that are connected to the Windows Update service but have no configured management policies governing updates. So, updates are managed by the Windows Update service. The majority are consumer devices, many of which are used on an irregular schedule, are often offline, and are difficult to update.
- Commercial devices that are self-managed and where the update process is not changed. These devices are actively managed by IT administrators using an on-prem infrastructure. Typically, this is based on Windows Server Update Services (WSUS), but update policies are not configured according to Microsoft best practices for a "cloud cadence" mentality.
- Commercial cloud-managed or self-managed devices with update process transformation. These devices are actively managed by IT administrators using Microsoft Endpoint Configuration Manager or Microsoft Intune (supported by Windows Update for Business), with update policies configured according to Microsoft best practices for a "cloud cadence" mentality.
Telemetry data from ten million devices was used to examine the average update adoption rate in these three segments from December 2020 to February 2021. The table below shows the results.
Device segment |
T+14 days |
T+28 days |
Microsoft Update managed |
73.2% |
78.8% |
Commercial self-managed, with no update process transformation |
55.0% |
80.0% |
Commercial cloud-managed or self-managed with update process transformation |
92.5% |
93.6% |
Only the systems that use an update management policy that meets Microsoft's ideas of "cloud cadence" are 92.5% up to date with patches after 14 days and 93.6% after 28 days. The table highlights the poor patch rate of self-managed devices, whose security update installation rate is a meager 55% only after 14 days. Even devices managed via Windows Update are still better at 73.2%.
Cloud cadence: patch as quickly as possible
Based on this evaluation, Frost explores the question of how do companies achieve a rapid update installation rate in the 14-day window of more than 90% for commercial devices managed via the cloud? The article then gives advice on how to achieve the:
- First, fundamentally commit to "cloud cadence." By "cloud cadence" mentality, Microsoft means streamlining the review and approval processes for installing security updates.
- Second, eliminate the often misunderstood policies that appear to optimize update behavior but actually slow down the update machinery.
To assist administrators in efforts to increase update velocity in the enterprise, Microsoft has released the Windows 10 Update Baseline, which provides recommended configurations and tuning guidance to maximize the installation of device updates in the shortest amount of time. In the article, Frost then expands on the importance of testing and then releasing updates quickly and what can be done to achieve timely release.
Collision with practice?
At this point, I would like to make a break – if you are interested, you can read the Microsoft employee's comments at your leisure. I am interested in how the whole thing is judged from a practical point of view. I assume that no administrator maliciously blocks the updates for installation. In this context, however, from my point of view, there are some questions or points of view that remain hidden in the Microsoft universe.
- Problem 1: Update quality; There is the "we have no problems" feedback from consultants and administrators (mostly from larger organizations) on my blog. I don't like to dismiss that out of hand. But there is also the faction of administrators and users who are just annoyed by update problems, tremble every month before the patchday and often have to suspend the update installation because of serious problems. This runs contrary to the above shiney happy update worldview. And unfortunately, it must also be noted that Microsoft repeatedly fails to effectively close reported vulnerabilities with the first patch. It often takes weeks to fix them.
- Problem 2: We don't have anything else to do; The above Microsoft approach is based on the idea that administrators have the time and knowledge to test preview updates monthly and then release the rollout promptly. There may be administrators in larger organizations who are only responsible for Windows updates and have enough time as well as experience to test. However, my impression is that there is a large percentage of administrators in corporate environments who simply do not have the time for major testing and are rushing after the update installation. The only strategy to get halfway through is: delay updates a bit, see if there is collateral damage or reported in blogs like here and only then install. If a mass hack becomes known via a closed vulnerability, it is important to act quickly and patch.
- Problem 3: We are digging in the wrong hole; The tenor of the Microsoft article – and often the recommendation of "those in the know" is "update immediately to close 0-days or freshly fixed vulnerabilities". Without issues 1 and 2, I would sign that immediately. And Yes, the Hafnium Exchange hack in Spring 2021 apparently proves this (in truth it was a 0-day that was already exploited before the security update was released). But is this really our problem, or are we digging in the wrong hole?
Regarding problem 3: Sometimes a look at the statistics is helpful to find out which vulnerabilities are being exploited in attacks. First, a look at the 133-page Rand report on the lifetime of a 0-day vulnerability, which shows the following.
- Exploits and their underlying vulnerabilities have a fairly high average "life expectancy" of 6.9 years.
- For a given population of zero-day vulnerabilities, about 5.7 percent will be discovered by an external entity after one year.
You simply won't be able to eliminate this risk with "even faster patching," because 0-day means the vulnerability is publicly known but there is no patch for it. This was exploited in the hafnium hack. The RAND report states that attackers use 0-days to attack companies and targets that are diligent about patching. Those that are not as diligent in patch management can be attacked via vulnerabilities that have been patched but not yet applied. 0-Day vulnerabilities are useful in some cyber operations (by criminals, militaries or Governments) are useful. But for mass attacks, there is a larger and probably more lucrative area that involves unpatched vulnerabilities in systems.
I had seen an article weeks ago that showed that the majority of attacks exploit known vulnerabilities for which updates have long been available. Unfortunately, I have forgotten the source. But while researching, I found the Comparitech article25+ cyber security vulnerability statistics and facts of 2021 . 75% of attacks in 2020 exploited vulnerabilities that were at least two years old. The source used is the Check Point Check Point Cyber Security Report 2021, which indicates that three out of four attacks exploited vulnerabilities reported in 2017 or earlier. And 18 percent of attacks exploited vulnerabilities disclosed in 2013 or earlier, which were at least seven years old.
Newly disclosed and patchable vulnerabilities from the same year accounted for only 5% of the attack surfaces. This makes me wonder: Are we diggingin deep in the wrong hole? Patching as quickly as possible, according to the Microsoft post above, is perhaps not so much the problem. If I can patch quickly, that's fine and to be welcomed. But the greater efficiency gain is to be bring unpatched systems with vulnerabilities known for years up to the current update status.
It is all right for Microsoft to propagate the "pure doctrine". But tunnel vision often leads nowhere. However, since I do not administer systems in a corporate environment, I am interested in the assessment of administrators in that area. Is the Microsoft approach feasible and helpful in practice for you? Or am I not completely wrong with my assessment of the problems? What works and what remains a pious wish for good weather periods? Your opinion from the circle of administrators is needed.
Advertising
Thanks for such a well-thought out article. MSFT's current fingers-in-ears "Lalalalalala.." attitude when it comes to admitting it's patches often wreak more havoc than the vulnerability itself would, is a typical example of corporate hubris that someday will bring it down.