What I understand is that desert and beach sand tends to be old/heavily weathered sand that is too fine with rounded edges. And is physically weak. Sea sand also tends to have chlorides which causes problems.
The foundation of the house I'm living in was probably made with beach sand (1908 construction). One wall that gets wet frequently is turning to dust.
Flip-side of that is the timbers are all old growth redwood. If you ever poke at the stuff after 100 years as long as it didn't get wet it's fine. 50-50 chance whoever buys this place will replace the foundation. And hopefully with proper concrete. Do that and the building will last into the next century.
The point is supposedly that providing housing for the current exploding population is more important/urgent than their great-grandchildren having to tear down and rebuild their houses in 100 years.
There is a beach in south Sweden which was mined for sand, I think it was for biosand filters for water purification. The said they exported sand to Saudi from there. I presume different sands have different properties.