This is not the case in some prominent housing markets (London & NYC come to mind). There are owners who purchase the properties as a way to store wealth in the US. These dwellings frequently sit empty most of the time.
I learned about it after a trip to OT301 with some friends from the CouchSurfing meetup in Amsterdam in 2013. Apparently the law has changed though, and now people are being criminally charged for squatting empty buildings.