Women needed to stay with the kids to breastfeed. Baby formula was only invented around 1865. Any traditional culture is going to need women to stay with the babies to keep them from starving.

However women would take babies with them while doing things like picking nuts and berries.