The opposite, as it is a privilege to work somewhere without externalities.

The privilege of ignorance? Because no such thing exists, unless of course you are ignorant of the externalities which is very likely.

As they say, ignorance is bliss.

I read it as the privilege of being able to choose a lower paying job (that lets you sleep better at night, be more fulfilled, etc)

It's a privilege because the number of roles with no externalities are limited