Port knocking is definitely dumb, but "increase your password lengths, or cryptographic key sizes" does nothing if your ssh binary is compromised and anyone can send a magic packet to let themselves in.
Strict firewalls, VPNs, and defense-in-depth is really the only answer here.
Of course, those things all go out the window too if your TCP stack itself is also compromised. Better to just air-gap.
Many people argue that a VPN+SSH is a reasonable solution, since it uses two separate implementations, where both are unlikely to be compromised at the same time. I would argue that the more reasonable option would be to split the SSH project in two; both of which validates the credentials of an incoming connection. This would be the same as a VPN+SSH but would not convolute the network topography, and would eliminate the need for two keys to be used by every connecting user.
However, in this case, the two-layer approach would not be a protection. Sure, in this case the SSH daemon was compromised, and a VPN before SSH would have protected SSH. But what if the VPN server itself was compromised? Remember that the SSH server was altered, not to allow normal logins, but to call system() directly. What if a VPN server had been similarly altered? This would not have protected SSH, since a direct system() call by the VPN daemon would have ignored SSH completely.
It is a mistake to look at this case and assume that since SSH was compromised this time, SSH must always be protected by another layer. That other layer might be the next thing to be compromised.
Best practice would have your internet exposed daemon (vpn or ssh) running a fairly locked down box that doesn’t also have your valuable “stuff” (whatever that is) on it.
So if someone cracks that box, their access is limited, and they still need to make a lateral move to access actual data.
In the case of this backdoor, if you have SSH exposed to the internet on a locked down jump box, AND use it as your internal mechanism for accessing your valuable boxes, you are owned, since the attacker can access your jump box and then immediately use the same vulnerability to move to an internal box.
In the case of a hypothetical VPN daemon vulnerability, they can use that to crack your jump box, but then still need another vulnerability to move beyond that. Not great, but a lot better than being fully owned.
You could certainly also accomplish a similar topology with two different SSH implementations.
>This would not have protected SSH, since a direct system() call by the VPN daemon would have ignored SSH completely.
I don't know how VPNs are implemented on Linux, but in principle it should be possible to sandbox a VPN server to the point where it can only make connections but nothing else. If capabilities are not enough, ebpf should be able to contain it. I suspect it will have full control over networking but that's still very different from arbitrary code execution.
That would be possible, yes, but it’s not the current situation. And since we can choose how we proceed, I would prefer my proposal, i.e. that the SSH daemon be split into two separate projects, one daemon handling the initial connection, locked-down like you describe, then handing off the authenticated connection to the second, “inner”, SSH daemon, which also does the authentication, using the same credentials as submitted by the connecting user. This way, the connecting user only has to have one key, and the network topology does not become unduly twisted.
Huh, I briefly looked at the documentation for UsePrivilegeSeparation option and it looks very similar to what you're describing. Interesting why it didn't prevent this attack.
Strict firewalls, VPNs, and defense-in-depth is really the only answer here.
Of course, those things all go out the window too if your TCP stack itself is also compromised. Better to just air-gap.