> BigCompany A/B testing has this unfortunate tendency to demonstrate positive results to whatever you are testing.
No, it doesn't. It tests which one is more positive, not which is is positive.
> Like the A/B test that repeatedly demonstrated that after you buy a blender, your most likely purchase is another blender... (which Amazon still hasn't fixed nearly a decade later)
> No, it doesn't. It tests which one is more positive, not which is is positive.
"positive" in the sense of "always answering questions with 'yes'", not in the sense of "positive numbers". Turns out when you start linking people's compensation to their datascience results, the results start to lean in whatever direction optimises for compensation...
> That isn't an A/B test.
A series of A/B tests. All the alternatives failed to produce better metrics than the status quo, so everyone became satisfied that their local minima was optimal
No, it doesn't. It tests which one is more positive, not which is is positive.
> Like the A/B test that repeatedly demonstrated that after you buy a blender, your most likely purchase is another blender... (which Amazon still hasn't fixed nearly a decade later)
That isn't an A/B test.