Hello, IT, have you tried turning it on and off again 15 times?
Seriously though - this entire outage is the poster child for why you NEVER have software that updates without explicit permission from a sysadmin. If I were in congress, I would make it illegal, it's an obvious national security issue.
To play devil’s advocate, a staged rollout for antivirus definitions somewhat defeats the point since those definitions are supposed to be constantly updated.
I agree with the rest, especially the use of a memory unsafe language to do parsing in the kernel by a billion dollar security company blows my mind.
How can you even run a security company without any security professionals reading your code even incidentally? An impressive level of incompetence.
At least they could make a in house playground in the process to see if their new version ever work. Maybe something like guest computer in public area. Or some sort of vm to emulate end user system to see if they ever boots. And somehow we still get this.
How the heck they didn't find out the new version prevent the computer from booting at all?
> Panicking when the file doesn’t parse because it’s not a memory safe language?
Whether a program panics or recovers when attempting to parse bad data is entirely orthogonal to memory safety. Do you have any in-depth technical information about the bug itself that you're basing this on?
Is it normal to make outbound connections during boot? Doesn't that circumvent a firewall? That seems like something a security team evaluating whether they want this software on their network might care about during an eval period.. right?
Looking at the contents of c:\windows\system32\drivers\crowdstrike suggests it does all sorts of weird shit right down to injecting itself into UEFI and futzing with firmware. It's literally in everything.
Unfortunately "security" folk these days are box ticking fuckwits and this product brief ticked all the boxes. They do not understand any more traditional methodologies other than "install these magic beans and action the reports".
Invest in better software and network architecture and DR strategy instead.
That's not the big no-no here. Lack of any real DRP is. Sure, it's cheaper to just buy CS Falcon (and who knows what other amazing vendors supplied timebombs are ticking silently) than paying sysadmins and developers ... and letting them build something that does what it needs, not much else, so there's no need to put these fantastic "single agents" from these RCE-as-a-service vendors on all the fucking servers.
What % of those sysadmins are then going to turn around and script something to auto-approve those updates, once they realize that they are A) requested at inconvenient times and B) are related to security?
Who's going to take the risk of appearing to have sat on an important update, while the org they support is ravaged by ThreatOfTheDay, because they thought they knew better than a multi-billion dollar, tops-in-their-field company?
(I'm not necessarily saying that's actually objectively correct, but I can't imagine that many folks are willing to risk the downside)
> why you NEVER have software that updates without explicit permission from a sysadmin
In general I agree, but this case is quite messy. It's more like your anti-virus had a bug since forever that if it loads a broken virus definition it bricks your system. And a broken virus definition finally happened today.
Do you want every virus definition (that is updated every few hours) to require explicit permission from a sysadmin?
You’re learning the wrong lesson here. Automatic security updates in Debian and Ubuntu actually get tested and work.
The RCE in ssh a week ago is an argument for enabling automatic security updates. (And for security in depth, putting everything behind VPN for example)
This example is probably an argument for not running windows on critical systems due to insufficient focus on security from the beginning which has lead to a need for things like crowdstrike.
They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.
>They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.
I wish people would stop making blanket statement as if they know how every company in the world runs. Plenty of Linux machines are running CS, and it's not only because they are forced to for compliance. NG AV has been picking up speed as a "just in case" thing for Linux and Mac for years now. Your anecdote does not apply to everyone.
I understand the logic of this but it is somewhat based on the assumption - which most industries have in droves - that people in THAT industry are the competent bullwhark against stupidity.
I consulted for a company for a while where the 'sysadmin' was the owner's mother - who bought laptops from walmart. Not only could she NOT have approved updates like this, even if she could she would have she wouldn't have had any knowledge whatsoever with which to make a determination if it worked.
In an abstraction, the problem really is with externalities. These approaches to updates exist because people who CAN'T do what you describe are likely a more dominant part of the threat model than this happening to people you do describe. The resulting fix, as we're seeing, is very reliable until it isn't...and if the isn't is enormous in scale the systems aren't setup to fail gracefully.
If you want to make a rule...require graceful failure.
What would the sysadmins do in this context? Read the release notes of the update? The only thing they would do is update and then be responsible for the problem, and in that case you're back to this exact problem.
It's not like they'd read the source code or examine every file that's been changed or downloaded for a proprietary kernel module for every crowdstrike update (there must be a LOT of them).
They would release the update in a testing/sandbox environment first before rolling out kernel-level changes to every computer on their network.
They're the same team who mandate you use a 3-year-old browser version and 5-year-old OS, because you can't be trusted to manage your own updates, so they do know the idea.
Would this have changed something for this specific problem? I usually 100% agree with you fwiw, I just don't think this would've helped here because it seems like an almost "non update"? Most people claim there has been no update to the software, and no prompt or option to update it or not
It's a file that was downloaded from Crowdstrike's servers, which have presumably been whitelisted in the firewall, and used to configure the software. Of course it's a software update, regardless of whether the file says .exe or .dll or .sys or .txt, and regardless of whether there was a prompt.
Again, the same team in most enterprises wouldn't dream of letting you have an auto- updating Firefox Nightly, they know how to configure software so it doesn't phone home for updates or is blocked from phoning home.
> It is illegal to use a metal detector to search for archaeological objects anywhere in the Republic of Ireland, unless you have written permission from the government.
> Those convicted of the offence can face a fine ranging to more than €63,000 (£53,000).
No wonder they donated anonymously, what an insane law.
>No wonder they donated anonymously, what an insane law.
By contrast, where I live in Philadelphia, developers are not required to perform any archaeological studies before excavating - even along the Delaware River waterfront, where the oldest European settlements are, as well as countless indigenous sites.
Sometimes, before history gets scraped away and sent to the dump, bottle diggers will excavate the trash pits, typically discarding anything that's not 100% intact, and selling the 18th and 19th century bottles on eBay or at flea markets. However, like the axe heads in the article, these artifacts are absent context, removing nearly all historical value. And of course, the stratification of the pits they're extracted from is also destroyed, further reducing the ability to interpret any finds that might otherwise have been saved.
It's only projects paid for by our federal government that are required to do archaeological studies, and when they do, it's not uncommon to find early colonial artifacts, but also remnants of pre-contact Lenni-Lenape sites.
You only really get one shot at recovering history through archaeology. That doesn't mean that preservation holds permanent veto over progress, but a little bit of disincentivization can go a long way in the study of history.
Systems like these need the right incentives to work.
If the builder is compensated for the costs associated with working around the fonds, plus a little extra, they will be happy to report everything they find.
I'd throw in making museums obliged to display which builder company found the item when displaying it.
This probably costs a lot more than the average government wants to spend on archeology though.
Sounds like you want to read through https://www.museum.ie/en-IE/Collections-Research/The-Law-on-... to tone down the indignation a little. It's a perfectly fine law: if you use a metal detector on your farm and you happen to find archeological metal, you're not breaking any laws. But now you do have a duty to report that find, and you can't keep using your metal detector and stabbing the ground with a shovel when it beeps because then you'd be intentionally disturbing and possibly destroying a site of potentially historical significance.
You don't lose "the complete use of your land" because a small part of it is considered historically significant, it's not the US. You do get to wait for them to figure out the boundary though.
As for your Wikipedia link, it's a good idea to actually read the whole text in it.
Again, it's not the US: are you just trying to come up with arguments that you can keep being angry about instead of knowing, or looking up, how this law's been used?
It's not, which is my point. Criticize it on the merits of how the law's used in Ireland instead of coming up with "things that happen" that don't happen.
'Lenihan himself mentioned in a 2009 interview that the National Road Authority "were forced to bend the road a little bit around the bush, which they did, which is fine. There was no need to demolish the bush at all. Just adjust the road a little bit and landscape the bush into the roadway"'
Is saying that you searched for non-archeological objects and just happened to find one a valid defense? Or is it illegal to find archeological objects even if you weren't explicitly searching for them?
We have a law that somehow covers this in France. It is permitted to use a metal detector for leisure provided that it is not a search for archaeological objects.
However, this law can be interpreted from many angles, as it is difficult to define what is archaeological or not. There are certain associations which defend “leisure detection” as long as it is not located on a registered archaeological or historical site. But their point of view is not shared by archaeologists, who consider that any place can be considered an archaeological site, relegating leisure detection to fine sandy beaches.
In this context, case law says that there are no completely accidental archaeological discoveries with a metal detector.
>The letter stated that the axe-heads were discovered in County Westmeath using a metal detector.
>The museum explained those "severe penalties" are in place because unauthorised metal detecting can cause "serious damage to Ireland’s archaeological heritage
It would be a lot more reassuring if they just fined the culprit 1 EUR and closed the case. Most European countries don't allow a judged case ("res iudicata") to be reopened unless new serious material evidence comes to light, and many not even then.
Yeah that law is ridiculous. What happens if you are a metal-detecting enthusiast looking for valuables and you stumble upon something archaeological? Are you just gonna get fined?
In my experience, nearly all metal-detecting enthusiasts hope to find valuable antiques, not modern iron scrap. Alright, some of them hope to find lost jewelry, but also valuable antiques. None of the ones I've known are prospecting for boring metal veins, for example.
There is a bit of a line to be drawn between 'valuable antiques' and things that are 'archeological' though. Lots of people are happy to find bullets and belt buckles from 100 years ago vs specifically looking for iron age artifacts.
I'm fairly certain that "anywhere" bit is wrong. You only need a license when doing it on public land, except for scheduled monuments, historic monuments, and areas of archaeological interest which require special permission as well.
If it's on private land and you have permission from the landowner, you don't need the license. You still need to report any treasure or archaeological finds to National Museums NI though.
Seriously though - this entire outage is the poster child for why you NEVER have software that updates without explicit permission from a sysadmin. If I were in congress, I would make it illegal, it's an obvious national security issue.