It probably doesn't apply here (and might well be country/region specific), but one thing I've heard from friends who've done similar has been some resentment/hostility related to jobs that are only people with previous money can afford to take.

I've got several friends that made a lot of money in tech/law before switching to more fullfilling jobs (things like furniture maker, barista, something music-related, plumber, museum curator, librarian).

One argument they've heard is that by taking jobs that aren't economically feasible to do without money, they're encouraging low pay for others in that field who don't have any other choice. One or two volunteer for free, and have heard complaints that volunteers have destroyed careers in certain sectors.