Agreed, but how do you know for certain they don't surreptitiously upload your contacts after you have granted permission?

Sounds like you are on iOS?