We either run experiments with higher traffic diverted to treatment (eg: 50% instead of 1%), or run it for a longer time. This is the reason why we need companies to have at least 2K monnthly users to run pilots with us.
Thanks! Cool idea on that front and in general; fun to see people taking copy seriously!
I am still a little fuzzy on how this might play out on a real site, since lots of monthly users doesn’t always mean lots of conversions (especially for one-time, high dollar transactions).
Let’s say I have a page that gets 100k unique visitors per month. I show them 5 different variants of a “nudge” widget. Some do better than others, but they all hover at <1% CTR.
While there may be a story to tell as far as winners and losers (e.g. mobile users converted to variant A at 2x the rate of desktop users), how do you confidently report a “winner” from what amounts to a few thousand conversions in a month?
