At least to me, a partial cause for that is the Scientology "church" in the 1970s and later [1]. When the IRS backed down due to their threats instead of hitting them with fines, that sent a very bad signal: if you're big and aggressive enough, we'll let you continue making a joke out of the tax code and focus on smaller fishes instead.
That awful track record has a lot to do with priorities driven by political talking points and who has the power to complain about the IRS.
The book _Perfectly Legal_ by David Cay Johnston has a good chapter on this. (The book as a whole has its ups and downs, but the history of IRS management I found particularly interesting.)
To be honest though the IRS has an awful track record of focusing on lower income people :(