This is just FUD, there is nothing dangerous in having an SSH server open to the internet that only allows key authentication. Sure, scanners will keep pinging it, but nobody is ever going to burn an ssh 0day on your home server.
A few years ago a vulnerable compression library almost got pushed out that major Linux distros linked their OpenSSH implementations to. That was caught by blind luck. I'm confident there's a lot more shit out there that we don't know about.
"opsec" includes well defined things like threat modeling, risk factors, and such. "Things I have seen" and vague "better safe than sorry" is not part of that.
