Performance isn't an issue for us, since the existing processes stay alive until we've healthchecked new processes. That it's encrypted alleviates some concerns, but given the number of deploys we do a day - on over 50 services and counting - I'm hesitant to have secrets go over the network so often.

That said, if you're in Azure, it's certainly the easiest way to go, and definitely not a bad approach.