>Getting rid of it would bring housing costs down.
You answered your own question. Housing costs coming down is a good thing for people who need housing, but it's not really a good thing for people who own valuable homes, and people who own valuable homes tend to have much more political power.
I feel that's mostly true, but there may be many other similar factors. If you make "affordable" housing ("for who?" is the correct question), then you have more people that can't afford to spend as much, while pushing out the people that spend more.
Someone else stated it simply as "undesirables", which I think is the quick summation of your statement and other similar reasoning. It all comes down to "I get that people need a place to live, but they can find that somewhere else" or "we've got a good thing going here (for me, the politician)".
Zoning usually gets decided on on a very local level. The people who show up to community meetings tend to be older, white home owners who don't want the value of their home to go down. Fear of "undesirables" of course also likely plays a role and when I've heard it in person, even that gets wrapped in fear of that bringing down home values and schools.
Perhaps more white owners show up in majority white neighborhoods, but what evidence is there that white owners disproportionately show up to these meetings? I assume you can provide it based on this comment.
Do they own valuable homes or do they own valuable land? I can see the land value increasing because it is now rezone-able and you can build multiple units worth on it.
You answered your own question. Housing costs coming down is a good thing for people who need housing, but it's not really a good thing for people who own valuable homes, and people who own valuable homes tend to have much more political power.