This is insane. The company I currently work for provides dinky forms for local cities and such, where the worst thing that could happen is that somebody will have to wait a day to get their license plates, and even we aren't this stupid.
I feel like people should have to go to jail for this level of negligence.
Maybe someone tried to backdoor Crowdstrike and messed up some shell code? It would fit and at this point we can't rule it out, but there is also no good reason to believe it. I prefer to assume incompetence over maliciousness.
>True for all systems, but AV updates are exempt from such policies. When there is a 0day you want those updates landing everywhere asap.
This is irrational. The risk of waiting for a few hours to test in a small environment before deploying a 0-day fix is marginal. If we assume the AV companies already spent their sweet time testing, surely most of the world can wait a few more hours on top of that.
Given this incident, it should be clear the downsides of deploying immediately at a global scale outweigh the benefits. The damage this incident caused might even be more than all the ransomware attacks combined. How long to take to do extra testing will depend on the specific organization, but I hope nobody will allow CrowdStrike trying to unilaterally impose a standard again.
I wonder if the move to hybrid estates (virtual + on prem + issued laptops etc) is the cause. Having worked in only on prem highly secure businesses no patches would be rolled out intra week without a testing cycle on a variety of hardware.
I consider it genuinely insane to allow direct updated from vendors like this on large estates. If you are behind a corporate firewall there is also a limit to the impact of discovered security flaws and thus reduced urgency in their dissemination anyway.
Most IT departments would not be patching all their servers or clients at the same time when Microsoft release updates. This is a pretty well followed standard practice.
For security software updates this is not a standard practice, I'm not even sure if you can configure a canary update group in these products? It is expected any updates are pushed ASAP.
For an issue like this though Crowdstrike should be catching it with their internal testing. It feels like a problem their customers should not have to worry about.
Their announcement (see Reddit for example) says it was a “content deployment” issue which could suggest it’s the AV definitions/whatever rather than the driver itself… so even if you had gradual rollout for drivers, it might not help!
I came to HN hoping to find more technical info on the issue, and with hundreds of comments yours is the first I found with something of interest, so thanks! Too bad there's no way to upvote it to the top.
In most appreciations of risk around upgrades in environments with which i am familiar, changing config/static data etc counts as a systemic update and is controlled in the same way
