It's pretty interesting that they didn't just introduce an RCE that anyone can exploit, it requires the attacker's private key. It's ironically a very security conscious vulnerability.
I suspect the original rationale is about preserving the longevity of the backdoor. If you blow a hole wide open that anyone can enter, it’s going to be found and shut down quickly.
If this hadn’t had the performance impact that brought it quickly to the surface, it’s possible that this would have lived quietly for a long time exactly because it’s not widely exploitable.
I agree that this is probably about persistence. Initially I thought the developer was playing the long-con to dump some crypto exchange and make off with literally a billion dollars or more.
But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event.
It's worth also noting that the spycraft involved a coordinated harassment campaign of the original maintainer, with multiple writing styles, to accelerate a transition of maintainership to the attacker:
At first I thought the guy who did this was a lone wolf but now I believe it was indeed state actor. They coordinated and harassed original maintainer into giving them access to the project, basically they hijacked the open source project. The poor guy(the original maintainer) was alone against state actor who was persistent with the goal of hijacking and then backdooring the open source project.
It seems like they were actively looking[0] which open source compression library they can inject with vulnerable code and then exploit and backdoor afterwards.
> At first I thought the guy who did this was a lone wolf but now I believe it was indeed state actor. They coordinated and harassed original maintainer into giving them access to the project, basically they hijacked the open source project. The poor guy(the original maintainer) was alone against state actor who was persistent with the goal of hijacking and then backdooring the open source project.
It could just as easily be a lone wolf capable of writing in different styles (really not difficult if one spends enough time online, one gets exposed to all sorts of styles) or even just your run of the mill criminal gang.
In fact, "multiaccount" cheating (where a single person has multiple accounts that interact with one another in a favourable manner, e.g. trading for stuff for reduced fees, or letting themselves be killed to pad stats, or expressing opinions leaving the impression of mass support, etc.) has been present in many MMO games for more than a decade. I recall an online game I was really into back in high school, eRepublik, which simulated real world politics, economics and warfare (in a really dumbed down version of course), and multiacounts were especially prevalent in politics, where candidates for election would often be padded by fake groups, or even entire countries being taken over by a gang of real people with like ten accounts each.
The complexity is nothing new. The only indicator this could be a state actor is the long con aspect of it. The attacker(s) were looking into years between starting and actually being able to exploit.
> “Recently I've worked off-list a bit with Jia Tan on XZ Utils and”
> "In 2021, JiaT75 submitted a pull request to the libarchive repository with the title ‘Added error text to warning when untaring with bsdtar’ which seemed legitimate at first glance. "
Reading that link, it seems like the vulnerability is that a file name gets printed, so you can add terminal control characters in a file name and have it printed.
Ugh, that this psyops sockpuppetry may have started or contributed to the maintainer's mental health issues seems like the most depressing part of all this. Maintaining OSS is hard enough.
I hope that one takeaway from this entire situation is that if you're a maintainer and your users are pushing you outside your levels of comfort, that's a reflection on them, not on you - and that it could be reflective of something far, far worse than just their impatience.
If you, as a maintainer, value stability of not only your software but also your own mental health, it is entirely something you can be proud of to resist calls for new features, scope increases, and rapid team additions.
Yeah, that was a hard read. I think it highlights the importance of emotionally distancing yourself from your projects.
It's also interesting because it exploits the current zeitgeist when it comes to maintainers' responsibilities to their users and communities. Personally, I think it puts too much expectation on maintainers' shoulders, especially when they're working for free.
Personally, I think maintainers are doing favors for users, and if they don't like how the project is progressing or not, then too bad. That's not a popular sentiment, though.
Probably didn't start them, considering that he already mentioned (long-term) mental health issues in the mailing list discussion in which the (likely) sock puppets started making demands.
But it's hard to see the whole thing helping, and it is some combination of depressing and infuriating. I hope he's doing ok.
I think the most convincing case made about the sock puppets is around account creation dates, and also people disappearing after they get what they need. Like Jigar disappearing after Jia becomes maintainer. Or the guy "misoeater19" who creates his debian bug tracker account to say that his work is totally blocked on needing xz 5.6.1 to be in debian unstable.
>But if that was the case they wouldn't bother with the key. It'd be a one-and-done situation. It would be a stop-the-world event.
Why not? It's possible someone else could've discovered the exploit before the big attack but decided not to disclose it. Or that they could've disclosed it and caused a lot of damage the attacker didn't necessarily want. And they easily could've been planning both a long-term way to access a huge swath of machines and also biding their time for a huge heist.
They have no reason to not restrict the backdoor to their personal use. And it probably is spycraft of some sort, and I think more likely than not it's a nation-state, but not necessarily. I could see a talented individual or group wanting to pull this off.
I think we need to consider the context. The attacker ultimately only had control over the lzma library. I'm skeptical that there's an innocent looking way that lzma could have in the open introduced an "accidental" RCE vuln that'd affect sshd. Of course I agree that they also wanted an explicit stealth backdoor for all the other reasons, but I don't think a plausibly deniable RCE or authentication bypass vuln would have even been possible.
Inserting a change like this as a one off would cause lots of scrutiny, which would probably get it detected. Instead, the bad actor spent years contributing to the project before dropping this.
So, while writing the exploit might be a couple of hours work, actually pulling it off is quite a bit more difficult.
I don't think they were using complexity as the reason for that assumption, but instead goals. Adding security doesn't require a nation state's level of resources, but it is a more attractive feature for a nation state that wants to preserve it over time and prevent adversaries from making use of it.
This makes sense in closed source products where you'll never get to audit the source for such exploits, but little sense in open source projects where anyone can audit it.
That's to say an enterprise router or switch would likely have secured exploits put there by corporate and national security agencies, whereas open source exploits would benefit from the probable deniability.
And on the contrary, creating a vulnerability that’s not identifiable to a limited attack group provides for a bit more deniability and anonymity. It’s hard to say which is more favorable by a nation-state actor.
An interesting angle: if this was somehow observed in use instead of getting discovered without observing use, there would be a glaring bright cui bono associated with what it was being used for.
Theoretically, I could even imagine that public key security motivated by some "mostly benign" actor who fell in love with the question "could I pull this off?" dreading the scenario of their future hacker superpower falling in the wrong hands. It's not entirely unthinkable that an alternative timeline where this was never discovered would only ever see this superbackdoor getting used for harmless pranks, winning bets and the like. In this unlikely scenario, the discovery, through the state actor and crime syndicate activities undoubtedly inspired by the discovery, would paradoxically lead to a less safe computing world than the alternative timeline where the person with the key had free RCE almost everywhere.
It'll be kind of tragic if this backdoor turns out to be the developer's pet "enable remote debugging" code, and they didn't mean for it to get out into a release. ;)
Not only did it mandate authentication, but there was also a Killswitch, and there's no other reason for that than because you have reasons to suspect someone may have access to this sensitive world-stopper exploit and use it against you. That leaves simply a (reasonably, but still) paranoid person, and people worried about that+consequences are unlikely to have the mental well-being to pull this off, or, a group of people.
If you think of it as a state sponsored attack it makes a lot of sense to have a "secure" vulnerability in system that your own citizens might use.
It looks like the whole contribution to xz was an effort to just inject that backdoor. For example the author created the whole test framework where he could hide the malicious payload.
Before he started work on xz, he made contribution to libarchive in BSD which created a vulnerability.
It did, actually: the filename can contain terminal control characters, which thanks to the change from safe_fprintf to fprintf, were printed without escaping, which allows the creator of the archive being extracted to control the terminal of the user extracting the archive.
(c) kind of contrived to get to execute; you have to somehow fail to extract the archive but on the happy path you'll see a bunch of weirdly-named files
I think it's distinctly higher-probability that this change was just meant to build credibility for the GitHub account. The diff is a fairly trivial patch to a minor issue filed around the same time as the pull request.
Yes, i think one thing we should learn from this is that suspicious code is suspicious code, and anyone asserting that some suspicious code cannot be exploited is suspicious themselves. I don't think we should inquisition half the industry, but i do think people should be a lot more careful about saying that one small exploitable thing definitely cannot be part of a larger exploit.
It's obvious, basically no one knows what's going on in the _vast_ majority of code running out systems these days. And even if you know 99% the attackers only need to be right once
If I read that correctly the problem is that it prints a filename that might include terminal control sequences that come from an attacker-controlled file name.
For real, it's almost like a state-sponsored exploit. It's crafted and executed incredibly well, the performance issue feels like pure luck it got found.
I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.
I doubt that if Google detected it with some internal tool, they'd reach out to Microsoft to hide their contribution.
It was reported by an MS engineer who happens to be involved in another OSS project. MS is doing business with the US intelligence community, for example there is the Skype story: First, rumors that NSA offers a lot of money for people who can break Skype's E2E encryption, then MS buys Skype, then MS changes Skype's client to not be E2E encrypted any more and to use MS servers instead of peer to peer, allowing undetectable wiretapping of arbitrary connections.
But it's a quite credible story too that it was just a random discovery. Even if it was the NSA, why would they hide that capability. It doesn't take much to run a script to compare git state with uploaded source tarballs in distros like Debian (Debian has separate tarballs for the source and the source with Debian patches applied).
> It was reported by an MS engineer who happens to be involved in another OSS project.
I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).
Thanks for the explanation. Also thanks for catching this and protecting us all; I think in the end it's way more believable that you indeed found it on your own, above was just brainless blathering into the ether =). Lastly, thanks for your Postgres contributions.
Intelligence agencies are very careful about sharing their findings, even with "friends", because the findings will disclose some information about their capabilities and possibly methods.
Let's say agency A has some scanning capability on open source software that detected this backdoor attempt by agency B. If they had gone public, agency B now knows they have this ability. So agency B will adjust their ways the next time and the scanning capability becomes less useful. While if agency A had told Microsoft to "find" this by accident, nobody would know about their scanning capability. And the next attempt by agency B would only try to avoid having the performance impact this first attempt had, probably leaving it visible to agency A.
Googling for "site:nsa.gov filetype:pdf" gives tens of thousands of results with documents produced by the NSA about various things (the google counter is known to lie but that's not my point). They do publish things.
He is also one of the sharpest developers I have had the fortune to speak to (and do some minor work with on the mailing list) and he knows a lot about performance and profiling. I 100% think that he found it on his own. He would also be an odd choice for Microsoft to pick since I doubt he works anywhere close to any of their security teams (unless they went through some serious lengths to find just the guy where it would be 100% believable that he just stumbled on it).
Yeah definitely, good point. I got that wrong above: He is in fact a postgres contributor who happens to be MS employee and not an MS employee who happens to be a postgres contributor.
The attacker changed the projects contact details at oss fuzz (an automated detection tool). There’s an interesting discussion as to whether that would have picked up the vulnerability https://github.com/google/oss-fuzz/issues/11760
I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz.
I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.
How many oss-fuzz packages have a Dockerfile that runs apt-get install liblzma-dev first?
Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well.
Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.
This is interesting, but do you think this would have aroused enough suspicion to find the backdoor (after every Ubuntu user was owned by it)? I don't see why this is the case. It wasn't a secret that ifuncs were being used in XZ.
And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this:
to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script.
With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.
I mean, yes. I’ve read all the commentary I can get my hands on about this incident because it is fascinating and this is the first instance of some parallel construction theory of finding it I’ve seen.
Second, maybe a routine dependency review is how they _actually_ found it but they don’t want future people like this focusing too much on that otherwise they may try to mitigate, whereas now they may focus on something inane like a 0.5 second increase in sshd’s load time or whatever.
Sorry for that ugly comparison, but that explanation reminds me of the theories when covid started, that it was created by secret organization that is actually ruling the world.
People love when there's some explanation that doesn't involve randomness, because with randomness looks like we don't have grasp on things.
Google actually had tooling that was detecting it, but he disabled check that would show it.
Google/Microsoft/NSA could just say they detected it with internal tooling and not disclose how exactly. Google and Microsoft would love to have credit.
While we are speculating, there was this case of university students trying to introduce malicious commits in the kernel in order to test open source to such attack vectors. Perhaps this was similar "research" by some students.
It's really interesting to think what might've happened if they could've implemented this with much less performance overhead. How long might it have lasted for? Years?
I don't think it was executed incredibly well. There were definitely very clever aspects but they made multiple mistakes - triggering Valgrind, the performance issue, using a `.` to break the Landlock test, not giving the author a proper background identity.
I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off.
Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.
Maybe it's the first successful attempt of a state which nobody would right now suspect as capable of carrying this out. Everyone is looking at the big guys but a new player has entered the game.
one question I still have is what exactly the performance issue was? I heard it might be related to enumeration of shared libraries, decoding of the scrambled strings[1], etc. anyone know for sure yet?
one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin.
I read someone speculating that the performance issue was intentional, so infected machines could be easily identified by an internet wide scan without arousing further suspicicion.
If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.
Very small differences in performance can be detected over the network as long as you have enough samples. Given that every port 22 is being hit by a gazillion attempts per day already, sample count shouldn’t be an issue.
So if distinguishing infected machines was their intention they definitely over-egged it.
I'm not sure why everyone is 100% sure this was a state-sponsored security breach. I agree that it's more likely than not state-sponsored, but I can imagine all sorts of other groups who would have an interest in something like this, organized crime in particular. Imagine how many banks or crypto wallets they could break into with a RCE this pervasive.
Especially considering this introduced a 500ms waiting time. But surely this was quite a risky time investment, 2 years. How likely is it that this was the only attempt if this was done by a group? (And maybe there were failed attempts after trying to take over maintenance of other packages?) Maybe really a very well-funded cybercrime group that can afford such moonshot endeavours or a state group that doesn't completely know yet what it's doing or isn't that well equipped (anymore?). I'm definitely curious about analysis of attribution
I'm not 100% sure - it could have been a criminal(s) or even a single motivated actor.
That said it's a lot of work - 2 years at least. It's an exploit that's so good that you'd have to use incredibly carefully - also because if/when it's discovered it's going to break everywhere.
I've read descriptions about how the NSA (and presumably other such agencies) and they're really careful. The first job is to make sure the target doesn't get confirmation that they are in fact a target. The second is that they always cover their tracks so the target doesn't know they were a target.
Criminals tend to do the first but almost never the second so a tool like this - while I'm sure they would love - isn't worth the amount of work it would take to develop.
Again - I'm not 100% on this but ... 40% ? say 20% criminals, 10% lone wolf?
Was the performance issue pure luck? Or was it a subtle bit of sabotage by someone inside the attacking group worried about the implications of the capability?
If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war".
It was a psyop to increase the scrutiny around OSS components.
Kidding. Mostly...
But given the amount of scrutiny folks are going to start putting into some supply chains... Probably cheaper to execute than most company's annual security awareness budgets cost.
Considering how difficult it might be (and identifiable) to attempt direct exploitation of this without being sure your target is vulnerable, it’s plausible the performance issue allowed for an identifiable delay in attempts. This might be useful in determining whether to attempt the exploit, with an auto-skip if it received a response in less than N milliseconds.
This is not that concept. That concept is no one but us can technically complete the exploit. Technical feasibility in that you need a supercomputer to do it, not protecting a backdoor with the normal cia triad
If they determine the vulnerability is only exploitable by the NSA for reasons
such as computational resources, budget, or skill set, they label it as NOBUS
and will not move to patch it, but rather leave it open to exploit against current
or future targets.
If (!) the NSA regards ssh keys as secure, then from that article it sounds like the NOBUS thing would fit.
You do realize that plenty of state actors have no qualms with physically torturing and killing people in the most gruesome ways possible, right? Online bullying is peanuts in that context.
Am I reading it correctly that the payload signature includes the target SSH host key? So you can't just spray it around to servers, it's fairly computationally expensive to send it to a host.
Not quite. It still looks vulnerable: an attacker A without the private key impersonates a victim server V and reports their host key. A careless attacker B with the key tries to attack A, but an ends up recovering a valid payload targeting V.
You can just treat the entire thing as opaque and proxy everything to the host you're trying to compromise; as soon as you have an exploit string for a given host you can just replay it.
Ssh does client authentication after handshake. The server is required to sign the handshake result with its private key, so you won't get past handshake if you are a server that claims to have a public key that you don't know the private key for.
E: see RFC 4253, sections 8 and 10, and RFC 4252 for corroboration
Huh, I had erroneously thought the exploit string was sent earlier in the connection, before the handshaking completed (note the "handshake failed" error in xzbot on successful exploit, and also the fact that no logging is done).
But you're right: we've verified the hostkey by the time we send the special certificate. So there's no way to effectively replay this without access to the server keys. My original comment is incorrect.
I'm actually surprised there's no logging at INFO or higher after this succeeds, given that openssh typically logs preauth connection closes. But I guess the crutch is that we never log connection opens and we only really log failures in handshaking, and it's not like the backdoor is going to go out of its way to log the fact that it opened itself...
The whole point of asymmetric-key is that a middleman can't do that. Even if you relayed the entire handshake, all traffic after that is just line noise to you unless you have the transmitter's private key. You can't read it or mangle it (well, you can mangle it, but the receiving party will know it was mangled in transit). The exploit string on the wire for that transmission won't work in the context of any other transmission sequence.
Yeah I had mistakenly thought the exploit string was transmitted during key exchange (read too quickly on "pre-auth"), which is incorrect; see sibling comment. I'm unfortunately past the edit window now.
I suppose the original attacker could give really fancy exploit strings that verify the eventual payload strongly enough to prevent someone replaying the attack from accomplishing much.
It's a (failed) case study in "what if we backdoor it in a way only good guys can use but bad guys can't?"
Computers don't know who's / what's good or bad. They're deterministic machines which responds to commands.
I don't know whether there'll be any clues about who did this, but this will be the poster child of "you can't have backdoors and security in a single system" argument (which I strongly support).
IMHO it's not that surprising; asymmetric crypto has been common in ransomware for a long time, and of course ransomware in general is based on securing data from its owner.
