"They" are not the cause of the shortage. You will have a housing shortage even without "Them" because of zoning issues and housing supply shortage.

It's both

Who should buy the housing stock then? Investment funds that will convert the homes to long-term Airbnbs?

More people wanting a product is a short-term price driver. If the price stays exorbitant over years and decades that is a supply-side problem that will not be fixed by slightly reducing the number of consumers.