For one thing, what you learned can stop working when you switch to a new model, or just a newer version of the “same” model.

All that means is that you verified the null hypothesis which should be that it doesn’t work

If you create hypothesis tests that are not written in or specific enough then you’re right you’re not gonna be able to do science

Incidentally 99.9% of people I know have no instinct for how to actually do science or have rigor or focus to actually do it in a way that is usable