But an attacker with one of the biggest vulnerabilities on earth (hell, ssh noauth 0day) would very likely use it against big cloud providers and infrastructure (isps and others) and not burn it on your home server! Keeping it reasonably up to date with your distro's cycle is probably enough for most people doing this home server thing.
So of course, as things always are with security this is a matter of risk assessment and understanding your attack surface, a server with only public key and maybe on a special port goes a very long way, add fail2ban on top and i'd say it's probably fine for quite a while.
But that does make me think... what if... a wormable noauth 0day like that on ssh or some other popular system... how fast could it replicate itself to form the biggest botnet.. how long would it take, to take over all visible linux servers on the internet (so that your little home box ends up being a target)?
I guess at that point you are limited by bandwidth, but since you can scale that with every compromised server... hope someone does the math on that one day!
Ipv4 is only 4 billion addresses. It doesn't actually take very long to just try all of them. If you're running a service exposed to the internet and it has a published exploitable vulnerability, it's just a matter of time before it gets exploited. (that said, that time does give a little buffer for patching)