I can’t imagine it’s too hard, I think password authentication is the key. Your user password is the same as your FileVault unlock password. I think that there’s a pre-unlock and post-unlock ssh session trick going on. The pre-unlock session just doesn’t have access to anything in the data volume and is able to use the provided password to unlock the data volume.

This would explain why it won’t work with ssh key authentication.