> people who are in the biggest house they are likely to own and expect to downsize

That's covered by the case of "property values go down everywhere"; you only have a problem if your property value goes down but the value of property you want to buy doesn't.

No, if you have a $1m house and you're intending to downsize to a $500k house, and they both lose half their value, that's $250k less in your pocket for retirement.