> The problem with that is then you need a mechanism that creates non-uniformly distributed mass.

The mechanism is gravity; and we have good observational evidence that the mass distribution of the universe is not uniform, at least at the scales we can observe (we can see galaxy clusters and voids).