It's all in our heads. Human species is obsessed with the opinions of others and they define us more than what we actually do. So ignoring downvotes (and upvotes!) is actually a good mental exercise.