>>it's that the esteem of people you don't admire doesn't matter.

Yup!

One of the best sayings I heard on the topic of doing anything potentially viewed as odd:

"Those who mind, don't matter, and those who matter don't mind"