Handling Patch Tuesday’s LSASS Memory Leak

Almost every admin lives with some sort of reservations, about that one patch that causes unexpected problems. It happens, not near as often as it used to, but it still happens. On March 12, 2024, Patch Tuesday brought updates for Windows Server 2012-2022 on Domain Controllers:

• • Windows Server 2022 – KB5035857 – [OS Build 20348.2340]
  • Windows Server 2019 – KB5035849 – [OS Build 17763.5576]
  • Windows Server 2016 – KB5035855 – [OS Build 14393.5786]
  • Windows Server 2012 R2 – KB5035885 – Monthly Rollup

Many woke up the next day to memory exhaustion issues caused by a memory leak in the LSASS (Local Security Authority Subsystem Service) process. Since LSASS is involved in multiple facets of authentication, this presented a very real and disruptive issue, and it seemed the more users the DC serviced, the faster the leak. In some cases the memory consumption was growing at a rate of gigabytes per hour!

Mitigation

Ironically the suggested temporary mitigation, regular reboots, was also the major reported symptom, which was, you guessed it, reboots due to a crashing LSASS process! This would have almost been a funny first occurrence of a self-mitigating update disaster, if randomly rebooting DCs could in any way be considered funny…

At first the only real other option was to uninstall the affected update, but since it, itself, was a combination of bug fixes, it was not the best plan either. So on March 22, 2024 Microsoft released KB5037422 as an out of band (OOB) update to address the actual root cause of this issue.

How to deploy the update

Since this is an OOB update, it will not appear in normal update channels, as it is meant to be a fix for affected systems on a per needed basis. This will also result in it not showing in the list of Action1 available updates to push either manually or automated.

But all is not lost, one of our Discord users (thank you and a big shout out to tjferreira!) created a script to make deploying this OOB update to fix the LSASS process easy. Following the link, you can find script and see how implementing OOB updates in Action1 is still a relatively easy task because of how Action1 makes patching more streamlined overall. Once you define an update process, all of the tools that make Action1 the premier patching solution, still help you get this out of the way and back to your day.

Applying the script with Action1:

1. Log in to the Action1 Platform or sign up for an account, the first 100 endpoints are free with no feature limitations.

Setup in minutes to reduce your cyber risks and costs:

CREATE ACCOUNT WATCH DEMO

2. On the left scroll down and select “Script Library” then choose “+ New Script”
3. Fill in the relevant information then choose “Next Step”
4. On the next screen, Ensure it reads PowerShell (Default) as language, then select all text in the text box and delete everything that is there by default, then copy in the text from Discord exactly as it is in the RAW content, do not modify it in any way. Then choose “Next Step”
It is recommended on the next screen you run this by a system “IN YOUR TEST ENVIRONMENT”
Be aware this step WILL run on the client and could potentially act in ways not yet understood, ensure whatever system is being tested on has proper backup recovery procedure.

If you need any assistance in implementing this in your environment, just reach out to us on Discord, Reddit, or direct to support. Someone at Action1 (or our wonderful community support!) is always willing to help.