You can also just have multiple keys? Most of my devices have an authorized_keys that gives access to a couple laptops and my phone (which runs the normal openssh client in termux), which solves a lot of these access problems.