I would have thought they could spin up headless Chrome instances to simply pull down, render, and then index websites. Apparently this is too resource intensive for them? I'm sure the idea has come up (there's no way I thought of this and they didn't).