There's another historical difference between Scotland and the US here: the US has a near-unlimited supply of wood, while Scotland was mostly bare of trees by the 18th century.

Much cheaper to build walls from local stone and save the imported wood for the floors and roofs.