The dairy cows that produced the milk you bought at the grocery store, in the USA anyway, were highly unlikely to be eating only grass growing on "infertile land" watered only by the rain. They are eating corn and grain grown on fertile land, probably with irrigation.

Is there even enough "infertile land" in the USA to produce grass to feed enough dairy cows for USA consumption? I doubt it.

So that argument still leads to drastic reduction in dairy consumption required, and in the USA anyway probably much higher dairy prices.