Correct, someone with physical access could run a MitM attack and steal your passphrase. I just find this extremely unlikely, so I honestly don't care.